Claire Duvallet

Claire Duvallet

STEM equity @ Science for America | formerly: data science @ Biobot Analytics, PhD @ MIT, Luce Scholar in Cambodia | public health data, microbiome, personalized medicine | gender and racial equity in STEM | she/hers

Uploading data to the SRA with AWS

1 minute read

Published:

ZOMG you can upload files to the SRA automatically via Amazon S3!! Here are notes for when I necessarily need to do this again or help someone in my lab do this again.

A note that these notes are basically the instructions that the SRA gives, so if you actually read the SRA’s notes and follow them you’ll figure it out too.

When you’re at the step where you need to submit files to the SRA (i.e. you’re on a page with a link like https://submit.ncbi.nlm.nih.gov/subs/sra/<submissionID>/files):

Follow their directions to generate a policy

Use the AWS Policy Generator with the following settings to create a Bucket Policy:

  • Policy type: S3 Bucket Policy
  • Effect: Allow
  • Principal: arn:aws:iam::228184908524:user/SA-SubmissionPortal-S3
  • AWS Service: Amazon S3
  • Actions:
    • s3:ListBucket
    • s3:GetObject
  • Amazon Resource Name (ARN): arn:aws:s3:::

Copy the policy text and attach it to a bucket. Note that this doesn’t mean you make a new policy through IAM, instead follow these directions (go to the bucket on aws.amazon.com, click on “Permissions”, and paste in the policy text).

If you get an error, you might need to also add arn:aws:s3:::<your bucket name>/* to the Resources on the policy.

AND THAT’S IT!

(JK getting the metadata all in order is a bit of pain, but you can also upload a tab-delimited file so it’s not that bad tbh…)

OMG THAT WAS SO EASY!

P.S. I double-checked with the SRA help desk, and they confirmed that once you’ve gotten the confirmation email from the SRA that your data has been successfully uploaded and processed, you can delete the bucket that you used for the upload.

You May Also Enjoy

Thank you for 18 years of DVDs, Netflix

Soon, Netflix will be canceling its DVD-by-mail program, the original service that helped Netflix crush Blockbuster and got us used to watching movies on-demand from the comfort of our homes before streaming was a thing. Perhaps not coincidentally, my dad cancelled my family’s subscription to the DVD service this winter. As my brother wisely put it upon hearing my dad’s news, “Netflix can finally stop buying physical DVDs now that their last customer cancelled!” ... Read more

Early startup employee lessons learned, part 4: adapting to your changing role

In Part 3 of this series, I wrote about strategies to build a team with a positive organizational culture. That post was about the team; this post is about you: as an early employee, how does your role change with the changing company and how can you gracefully ride that wave? When I joined my company as the fifth person, I knew that if all went well and our company succeeded, I’d have a very interesting path within the organization. What I didn’t know was any of the details of what that meant, and importantly what skills I’d need to gracefully walk that path. This post is about the things I’ve learned over the past two years as we’ve grown from 5 to 100 employees, and as my role has undergone countless transformations in the process of growing our team and responding to the company’s needs. ... Read more

Early startup employee lessons learned, part 3: building culture

A core function of being an early employee at a rapidly growing startup is helping it grow! At my current job, I’ve helped the data science team grow from being just one person doing a little bit of everything (me!) to a team of over 15 people working in three focused sub-teams. Growing the data science team has been one of the best parts of my hyper-growth experience, and I’ve learned a lot of really useful lessons from it. ... Read more

The Boston morning commute time warp

Like many young adults our age, my partner and I did the classic pandemic move of fleeing the city and moving in with his parents. That’s how I discovered I actually really enjoy living in rural New Hampshire, and last December we officially moved in to our own place in southern New Hampshire. ... Read more