Spikee

Working with spikee results and datasets using jq and coreutils.

Run these from the main spikee workspace folder

Result files are JSONL. For the jq filters used here, see also Select With Key and Select Fields and Filter by Substring.

Find Successes in the Results Folder

cat results/FILENAME-* | jq -c 'select(.success)'

Get Result Lines Where an Error Happened

error is null when there was no error.

jq -c 'select(.error)' RESULT-FILE

Search Results by Response or Error Text

Matches lines where the response field contains one string, or error contains another. Useful for searching a results file for particular items.

jq -c 'select(((.response | strings) | test("some string in the response")) or ((.error | strings) | test("a specific error string")))' RESULTS-FILE

Get Items to Re-Run

1. Find all items with errors.
2. Extract their id fields into an ids file.
3. Use the ids file to pick those items out of the original dataset.
4. Save them as a new dataset.

# one id per line
jq -cr 'select(.error) | .id' RESULT-FILE | sort -uV > FAILED-IDS-FILE
 
# new dataset of only the failed ids
jq -c --rawfile f FAILED-IDS-FILE \
        '($f
         | split("\n")
         | map(select(length > 0))
         | INDEX(.)
        ) as $ids
        | select($ids[.id | tostring] != null)
    ' ORIGINAL-DATASET-FILE > NEW-DATASET-FILE

Split a Dataset into Chunks for Parallel Runs

Splits a dataset into 3 chunks on line boundaries, producing SPLIT-DATASET-CHUNK00.jsonl, SPLIT-DATASET-CHUNK01.jsonl and SPLIT-DATASET-CHUNK02.jsonl, so they can be run in parallel.

split -n l/3 -d --additional-suffix=.jsonl datasets/DATASET-TO-SPLIT datasets/SPLIT-DATASET-CHUNK