View a markdown version of this page

Mengimpor set baca ke a HealthOmics toko urutan - AWS HealthOmics

Terjemahan disediakan oleh mesin penerjemah. Jika konten terjemahan yang diberikan bertentangan dengan versi bahasa Inggris aslinya, utamakan versi bahasa Inggris.

Mengimpor set baca ke a HealthOmics toko urutan

Setelah Anda membuat penyimpanan urutan, buat pekerjaan impor untuk mengunggah kumpulan baca ke penyimpanan data. Anda dapat mengunggah file Anda dari bucket Amazon S3, atau Anda dapat mengunggah langsung dengan menggunakan operasi API sinkron. Bucket Amazon S3 Anda harus berada di Wilayah yang sama dengan penyimpanan urutan Anda.

Anda dapat mengunggah kombinasi kumpulan baca yang selaras dan tidak selaras ke penyimpanan urutan Anda, namun, jika salah satu set baca dalam impor Anda disejajarkan, Anda harus menyertakan genom referensi.

Anda dapat menggunakan kembali kebijakan akses IAM yang Anda gunakan untuk membuat penyimpanan Referensi.

Topik berikut menjelaskan langkah-langkah utama yang Anda ikuti untuk mengimpor kumpulan baca ke penyimpanan urutan Anda dan kemudian mendapatkan informasi tentang data yang diimpor.

Unggah file ke Amazon S3

Contoh berikut menunjukkan cara memindahkan file ke bucket Amazon S3 Anda.

aws s3 cp s3://1000genomes/phase1/data/HG00100/alignment/HG00100.chrom20.ILLUMINA.bwa.GBR.low_coverage.20101123.bam s3://your-bucket aws s3 cp s3://1000genomes/phase3/data/HG00146/sequence_read/SRR233106_1.filt.fastq.gz s3://your-bucket aws s3 cp s3://1000genomes/phase3/data/HG00146/sequence_read/SRR233106_2.filt.fastq.gz s3://your-bucket aws s3 cp s3://1000genomes/data/HG00096/alignment/HG00096.alt_bwamem_GRCh38DH.20150718.GBR.low_coverage.cram s3://your-bucket aws s3 cp s3://gatk-test-data/wgs_ubam/NA12878_20k/NA12878_A.bam s3://your-bucket

Sampel BAM dan CRAM yang digunakan dalam contoh ini memerlukan referensi genom yang berbeda, Hg19 dan Hg38. Untuk mempelajari lebih lanjut atau mengakses referensi ini, lihat Referensi Genom Luas dalam Registri Data Terbuka pada AWS.

Membuat file manifes

Anda juga harus membuat file manifes di JSON untuk memodelkan pekerjaan impor di import.json (lihat contoh berikut). Jika Anda membuat penyimpanan urutan di konsol, Anda tidak perlu menentukan sequenceStoreId atau roleARN, jadi file manifes Anda dimulai dengan sources masukan.

API manifest

Contoh berikut mengimpor tiga set baca dengan menggunakan API: satu FASTQ, satu BAM, dan satu CRAM.

{ "sequenceStoreId": "3936421177", "roleArn": "arn:aws:iam::555555555555:role/OmicsImport", "sources": [ { "sourceFiles": { "source1": "s3://amzn-s3-demo-bucket/HG00100.chrom20.ILLUMINA.bwa.GBR.low_coverage.20101123.bam" }, "sourceFileType": "BAM", "subjectId": "mySubject", "sampleId": "mySample", "referenceArn": "arn:aws:omics:us-west-2:555555555555:referenceStore/0123456789/reference/0000000001", "name": "HG00100", "description": "BAM for HG00100", "generatedFrom": "1000 Genomes" }, { "sourceFiles": { "source1": "s3://amzn-s3-demo-bucket/SRR233106_1.filt.fastq.gz", "source2": "s3://amzn-s3-demo-bucket/SRR233106_2.filt.fastq.gz" }, "sourceFileType": "FASTQ", "subjectId": "mySubject", "sampleId": "mySample", // NOTE: there is no reference arn required here "name": "HG00146", "description": "FASTQ for HG00146", "generatedFrom": "1000 Genomes" }, { "sourceFiles": { "source1": "s3://amzn-s3-demo-bucket/HG00096.alt_bwamem_GRCh38DH.20150718.GBR.low_coverage.cram" }, "sourceFileType": "CRAM", "subjectId": "mySubject", "sampleId": "mySample", "referenceArn": "arn:aws:omics:us-west-2:555555555555:referenceStore/0123456789/reference/0000000001", "name": "HG00096", "description": "CRAM for HG00096", "generatedFrom": "1000 Genomes" }, { "sourceFiles": { "source1": "s3://amzn-s3-demo-bucket/NA12878_A.bam" }, "sourceFileType": "UBAM", "subjectId": "mySubject", "sampleId": "mySample", // NOTE: there is no reference arn required here "name": "NA12878_A", "description": "uBAM for NA12878", "generatedFrom": "GATK Test Data" } ] }
Console manifest

Kode contoh ini digunakan untuk mengimpor satu set baca dengan menggunakan konsol.

[ { "sourceFiles": { "source1": "s3://amzn-s3-demo-bucket/HG00100.chrom20.ILLUMINA.bwa.GBR.low_coverage.20101123.bam" }, "sourceFileType": "BAM", "subjectId": "mySubject", "sampleId": "mySample", "name": "HG00100", "description": "BAM for HG00100", "generatedFrom": "1000 Genomes" }, { "sourceFiles": { "source1": "s3://amzn-s3-demo-bucket/SRR233106_1.filt.fastq.gz", "source2": "s3://amzn-s3-demo-bucket/SRR233106_2.filt.fastq.gz" }, "sourceFileType": "FASTQ", "subjectId": "mySubject", "sampleId": "mySample", "name": "HG00146", "description": "FASTQ for HG00146", "generatedFrom": "1000 Genomes" }, { "sourceFiles": { "source1": "s3://your-bucket/HG00096.alt_bwamem_GRCh38DH.20150718.GBR.low_coverage.cram" }, "sourceFileType": "CRAM", "subjectId": "mySubject", "sampleId": "mySample", "name": "HG00096", "description": "CRAM for HG00096", "generatedFrom": "1000 Genomes" }, { "sourceFiles": { "source1": "s3://amzn-s3-demo-bucket/NA12878_A.bam" }, "sourceFileType": "UBAM", "subjectId": "mySubject", "sampleId": "mySample", "name": "NA12878_A", "description": "uBAM for NA12878", "generatedFrom": "GATK Test Data" } ]

Atau, Anda dapat mengunggah file manifes dalam format YAML.

Memulai pekerjaan impor

Untuk memulai pekerjaan impor, gunakan yang berikut ini AWS CLI perintah.

aws omics start-read-set-import-job --cli-input-json file://import.json

Anda menerima tanggapan berikut, yang menunjukkan penciptaan pekerjaan yang berhasil.

{ "id": "3660451514", "sequenceStoreId": "3936421177", "roleArn": "arn:aws:iam::111122223333:role/OmicsImport", "status": "CREATED", "creationTime": "2022-07-13T22:14:59.309Z" }

Memantau pekerjaan impor

Setelah pekerjaan impor dimulai, Anda dapat memantau perkembangannya dengan perintah berikut. Dalam contoh berikut, ganti sequence store id dengan ID penyimpanan urutan Anda, dan ganti job import ID dengan ID impor.

aws omics get-read-set-import-job --sequence-store-id sequence store id --id job import ID

Berikut ini menunjukkan status untuk semua pekerjaan impor yang terkait dengan ID penyimpanan urutan yang ditentukan.

{ "id": "1234567890", "sequenceStoreId": "1234567890", "roleArn": "arn:aws:iam::111122223333:role/OmicsImport", "status": "RUNNING", "statusMessage": "The job is currently in progress.", "creationTime": "2022-07-13T22:14:59.309Z", "sources": [ { "sourceFiles": { "source1": "s3://amzn-s3-demo-bucket/HG00100.chrom20.ILLUMINA.bwa.GBR.low_coverage.20101123.bam" }, "sourceFileType": "BAM", "status": "IN_PROGRESS", "statusMessage": "The job is currently in progress." "subjectId": "mySubject", "sampleId": "mySample", "referenceArn": "arn:aws:omics:us-west-2:111122223333:referenceStore/3242349265/reference/8625408453", "name": "HG00100", "description": "BAM for HG00100", "generatedFrom": "1000 Genomes", "readSetID": "1234567890" }, { "sourceFiles": { "source1": "s3://amzn-s3-demo-bucket/SRR233106_1.filt.fastq.gz", "source2": "s3://amzn-s3-demo-bucket/SRR233106_2.filt.fastq.gz" }, "sourceFileType": "FASTQ", "status": "IN_PROGRESS", "statusMessage": "The job is currently in progress." "subjectId": "mySubject", "sampleId": "mySample", "name": "HG00146", "description": "FASTQ for HG00146", "generatedFrom": "1000 Genomes", "readSetID": "1234567890" }, { "sourceFiles": { "source1": "s3://amzn-s3-demo-bucket/HG00096.alt_bwamem_GRCh38DH.20150718.GBR.low_coverage.cram" }, "sourceFileType": "CRAM", "status": "IN_PROGRESS", "statusMessage": "The job is currently in progress." "subjectId": "mySubject", "sampleId": "mySample", "referenceArn": "arn:aws:omics:us-west-2:111122223333:referenceStore/3242349265/reference/1234568870", "name": "HG00096", "description": "CRAM for HG00096", "generatedFrom": "1000 Genomes", "readSetID": "1234567890" }, { "sourceFiles": { "source1": "s3://amzn-s3-demo-bucket/NA12878_A.bam" }, "sourceFileType": "UBAM", "status": "IN_PROGRESS", "statusMessage": "The job is currently in progress." "subjectId": "mySubject", "sampleId": "mySample", "name": "NA12878_A", "description": "uBAM for NA12878", "generatedFrom": "GATK Test Data", "readSetID": "1234567890" } ] }

Temukan file urutan yang diimpor

Setelah pekerjaan selesai, Anda dapat menggunakan daftar baca-set Operasi API untuk menemukan file urutan yang diimpor. Dalam contoh berikut, ganti sequence store id dengan ID penyimpanan urutan Anda.

aws omics list-read-sets --sequence-store-id sequence store id

Anda menerima tanggapan berikut.

{ "readSets": [ { "id": "0000000001", "arn": "arn:aws:omics:us-west-2:111122223333:sequenceStore/01234567890/readSet/0000000001", "sequenceStoreId": "1234567890", "subjectId": "mySubject", "sampleId": "mySample", "status": "ACTIVE", "name": "HG00100", "description": "BAM for HG00100", "referenceArn": "arn:aws:omics:us-west-2:111122223333:referenceStore/01234567890/reference/0000000001", "fileType": "BAM", "sequenceInformation": { "totalReadCount": 9194, "totalBaseCount": 928594, "generatedFrom": "1000 Genomes", "alignment": "ALIGNED" }, "creationTime": "2022-07-13T23:25:20Z" "creationType": "IMPORT", "etag": { "algorithm": "BAM_MD5up", "source1": "d1d65429212d61d115bb19f510d4bd02" } }, { "id": "0000000002", "arn": "arn:aws:omics:us-west-2:111122223333:sequenceStore/0123456789/readSet/0000000002", "sequenceStoreId": "0123456789", "subjectId": "mySubject", "sampleId": "mySample", "status": "ACTIVE", "name": "HG00146", "description": "FASTQ for HG00146", "fileType": "FASTQ", "sequenceInformation": { "totalReadCount": 8000000, "totalBaseCount": 1184000000, "generatedFrom": "1000 Genomes", "alignment": "UNALIGNED" }, "creationTime": "2022-07-13T23:26:43Z" "creationType": "IMPORT", "etag": { "algorithm": "FASTQ_MD5up", "source1": "ca78f685c26e7cc2bf3e28e3ec4d49cd" } }, { "id": "0000000003", "arn": "arn:aws:omics:us-west-2:111122223333:sequenceStore/0123456789/readSet/0000000003", "sequenceStoreId": "0123456789", "subjectId": "mySubject", "sampleId": "mySample", "status": "ACTIVE", "name": "HG00096", "description": "CRAM for HG00096", "referenceArn": "arn:aws:omics:us-west-2:111122223333:referenceStore/0123456789/reference/0000000001", "fileType": "CRAM", "sequenceInformation": { "totalReadCount": 85466534, "totalBaseCount": 24000004881, "generatedFrom": "1000 Genomes", "alignment": "ALIGNED" }, "creationTime": "2022-07-13T23:30:41Z" "creationType": "IMPORT", "etag": { "algorithm": "CRAM_MD5up", "source1": "66817940f3025a760e6da4652f3e927e" } }, { "id": "0000000004", "arn": "arn:aws:omics:us-west-2:111122223333:sequenceStore/0123456789/readSet/0000000004", "sequenceStoreId": "0123456789", "subjectId": "mySubject", "sampleId": "mySample", "status": "ACTIVE", "name": "NA12878_A", "description": "uBAM for NA12878", "fileType": "UBAM", "sequenceInformation": { "totalReadCount": 20000, "totalBaseCount": 5000000, "generatedFrom": "GATK Test Data", "alignment": "ALIGNED" }, "creationTime": "2022-07-13T23:30:41Z" "creationType": "IMPORT", "etag": { "algorithm": "BAM_MD5up", "source1": "640eb686263e9f63bcda12c35b84f5c7" } } ] }

Dapatkan detail tentang kumpulan baca

Untuk melihat detail selengkapnya tentang kumpulan baca, gunakan GetReadSetMetadata Operasi API. Dalam contoh berikut, ganti sequence store id dengan ID penyimpanan urutan Anda, dan ganti read set id dengan ID set baca Anda.

aws omics get-read-set-metadata --sequence-store-id sequence store id --id read set id

Anda menerima tanggapan berikut.

{ "arn": "arn:aws:omics:us-west-2:123456789012:sequenceStore/2015356892/readSet/9515444019", "creationTime": "2024-01-12T04:50:33.548Z", "creationType": "IMPORT", "creationJobId": "33222111", "description": null, "etag": { "algorithm": "FASTQ_MD5up", "source1": "00d0885ba3eeb211c8c84520d3fa26ec", "source2": "00d0885ba3eeb211c8c84520d3fa26ec" }, "fileType": "FASTQ", "files": { "index": null, "source1": { "contentLength": 10818, "partSize": 104857600, "s3Access": { "s3Uri": "s3://accountID-sequence store ID-ajdpi90jdas90a79fh9a8ja98jdfa9jf98-s3alias/592761533288/sequenceStore/2015356892/readSet/9515444019/import_source1.fastq.gz" }, "totalParts": 1 }, "source2": { "contentLength": 10818, "partSize": 104857600, "s3Access": { "s3Uri": "s3://accountID-sequence store ID-ajdpi90jdas90a79fh9a8ja98jdfa9jf98-s3alias/592761533288/sequenceStore/2015356892/readSet/9515444019/import_source1.fastq.gz" }, "totalParts": 1 } }, "id": "9515444019", "name": "paired-fastq-import", "sampleId": "sampleId-paired-fastq-import", "sequenceInformation": { "alignment": "UNALIGNED", "generatedFrom": null, "totalBaseCount": 30000, "totalReadCount": 200 }, "sequenceStoreId": "2015356892", "status": "ACTIVE", "statusMessage": null, "subjectId": "subjectId-paired-fastq-import" }

Unduh file data set baca

Anda dapat mengakses objek untuk kumpulan baca aktif menggunakan Amazon S3 GetObject Operasi API. URI untuk objek dikembalikan di GetReadSetMetadata Tanggapan API. Untuk informasi selengkapnya, lihat Mengakses HealthOmics baca set dengan URI Amazon S3.

Atau, gunakan HealthOmics GetReadSet Operasi API. Anda dapat menggunakan GetReadSet untuk mengunduh secara paralel dengan mengunduh masing-masing bagian. Bagian ini mirip dengan bagian Amazon S3. Berikut ini adalah contoh cara mengunduh bagian 1 dari kumpulan baca. Dalam contoh berikut, ganti sequence store id dengan ID penyimpanan urutan Anda, dan ganti read set id dengan ID set baca Anda.

aws omics get-read-set --sequence-store-id sequence store id --id read set id --part-number 1 outfile.bam

Anda juga dapat menggunakan HealthOmics Transfer Manager untuk mengunduh file untuk HealthOmics referensi atau set baca. Anda dapat mengunduh HealthOmics Manajer Transfer kemari. Untuk informasi selengkapnya tentang menggunakan dan mengatur Transfer Manager, lihat GitHub Repositori.