Questions tagged [amazon-web-services]

Amazon Web Services (AWS) is a collection of commercial services centered around virtualized, commodity and clustered computing, networking, database and storage, as well as supporting tools that are offered through infrastructure as a service.

Filter by
Sorted by
Tagged with
4 votes
2 answers
126 views

Reading S3 files from two different buckets but adding it to one list variable

I have a below code which reads all the S3 files from a particular S3 bucket and then it adds all those files in a ...
  • 43
0 votes
0 answers
43 views

Copy AWS S3 objects into DB

I have written a code which fetched the S3 objects from AWS s3 using S3 sdk and stores the same in our DB, the only problem is the task is repeated for three different services, the only thing is ...
  • 11
0 votes
0 answers
74 views

Go efficiently retrieve and process large volume of AWS s3 objects

Like the title says, code below tries to efficiently grab and handle a large number (1m+) of objects from an aws s3 bucket. Right now on a 10gb lambda function it can handle ~2m objects in ~10s End ...
  • 1,037
2 votes
1 answer
59 views

Bash script to take user input to set AWS envrionment variable the first time and automatically in subsequent calls

Background Our team has a utility called envmgr to set aws iam roles. The credentials are only good for 4 hours or so. As such, I have a number of scripts that call the command ...
  • 311
2 votes
1 answer
163 views

Read bunch of files in parallel and populate concurrent map

I am trying to read bunch of S3 parquet files in parallel from a S3 bucket. After reading all these files, I am populating my products and ...
  • 219
5 votes
1 answer
236 views

Rust CLI tool to synchronize files to S3

Learning Rust, I've written a small CLI tool which will Fetch the existing data in S3 Iterate through local folders Upload the files which don't exist in S3 I'd be keen to hear anything I might have ...
7 votes
2 answers
472 views

A Hit Counter For AWS Python Lambda powered with DynamoDB

I'm learning how to use DynamoDB and various AWS services, so I decided to write a simple hit counter application for my education of how to use DynamoDB. This application deduplicates hits from the ...
  • 875
1 vote
1 answer
74 views

Archiving MongoDB data to S3, with retryable steps

I have a python script to archive some data (export from mongo to a local json file and then upload it to s3) Is there a better way to implement/improve the current flow? Each step should be be ...
  • 203
3 votes
1 answer
207 views

Delete older EBS Volume snapshots except few with certain tags using Python, Lambda

I am learning Python and managed to accomplish the above. But my script doesn't seem optimal. Could someone help to take a review and suggest for betterment? ...
2 votes
1 answer
57 views

PHP filesystem on a shared file system

I have a project which we need to migrate from an on-prem solution to the AWS cloud. Currently, the software is running on a single instance. It's working fine but the business want the software to be ...
1 vote
1 answer
50 views

Cleaning Netflix-style recommendation engine datasets

I am working on a portfolio project that deals with creating a Netflix-style recommendation engine for movies. This code is currently ran locally but I will need to upload it into Jupyter Notebooks ...
1 vote
0 answers
318 views

Golang DB connection with AWS SSM Parameters and context

I'm working on setting up an app backend that's deployed automatically using infrastructure-as-code and a ci/cd pipeline. As a result of that I need the app server to gather DB credentials ...
  • 1,037
3 votes
0 answers
73 views

Bash script for rebalancing weighted round robin

I have written a tool that is used to rebalance weights in a weighted round robin in AWS. I am quite happy with it whereas I would welcome feedback from any Bash programmers. ...
4 votes
0 answers
2k views

Copying multiple S3 files in parallel using aws shell command

Any improvements to this welcome, I have got it working and I'm happy with it. I'm not really all that proficient with bash shell scripts though. Problem: AWS can copy multiple files pretty much as ...
  • 143
4 votes
1 answer
59 views

scrape AWS ECS and store summary in S3

This is my scraper for AWS ECS (Elastic Container Service). My primary use for this is to grep through the resulting file to see which cluster a given service is ...
  • 2,847
5 votes
1 answer
520 views

Designing a Simple Class that caches files from AWS

Background I have a pipeline that I run and it spins up multiple EC2 instances that process many jobs. I noticed that in many parts of my pipeline i am downloading files from amazon s3 multiple times. ...
  • 387
2 votes
0 answers
28 views

Code and Inputs on to use AWS temporary Credentials

I am adding below code for review as i am bit new to boto3. This Script adds tags (key and value) to All Volumes of EC2 listed in Excel. Is there any more effective way i could have written this? Also ...
  • 21
3 votes
1 answer
62 views

Python AWS Lambda, why did updating in place instead of creating and appending to a new list not decrease memory usage?

I have a lambda function that ingests a CSV, reads it as a list of dicts, modifies them (updating the Keys, adding a few values) and submits them to a firehose. Initially I had the following code: <...
2 votes
1 answer
107 views

Send Data from DynamoDB to Lambda (C#) and to Azure Queue

I am a beginner and trying to send data from AWS DynamoDB to Azure Queues. Note that this code will be invoked 10,000 and alot more. Can you guys review it once. ...
3 votes
0 answers
110 views

SageMakerTrainingOperator in Airflow

I have made an operator (surrounded by others operators) for training a model in sagemaker in airflow and I have doubts how would it be more readable or more pythonic. First I did this: ...
3 votes
1 answer
2k views

Exporting data from PostgreSQL as CSV to S3 bucket

I have a general question regarding DRY. As you can see there are several parts such as io.StringIO(), csv.writer, ...
1 vote
0 answers
97 views

AWS plugin to create a CloudFormation stack if it doesn't exist

The following Python class is what I am trying to refactor in the case where I have to add validation of the existence of an AWS cloudformation stack before creating it in the method create_cfn_stack :...
1 vote
1 answer
57 views

Enumerating AWS instances in Rails, with error handling

I have created a method, which is raising the same exception in case of any error: ...
1 vote
0 answers
72 views

AWS Lambda stack definition YAML

Design My desired outcome is as follows: S3 event triggers Lambda Function startTextractStateMachine Lambda Function ...
3 votes
0 answers
261 views

User registration in Actix w/ DynamoDB

I am trying to puzzle out api based user registration for a service I am building, but I have a strong feeling that my code is not optimal. It feels like passing "state" in the way I did prevents the ...
1 vote
1 answer
399 views

Python functions to save JSON and H5 to local files and S3

I'm trying to copy files to S3 after writing them locally: I have 2 functions which I need to do this. What is the best way to refactor the code that exists in both. ...
  • 601
3 votes
0 answers
95 views

Validating Amazon CloudFormation templates using Rspec

I have created an open source project that uses Rspec to validate the formatting of AWS::CloudFormation::Init YAML blocks in CloudFormation templates.1 My project is here. Some general concerns: Is ...
4 votes
1 answer
703 views

Running AWS commands works ok, but the code seems messy and I don't like the time.sleep bit

This is what I've come up with for a class that handles parsing AWS Textract output. So far it only gets the AWS Textract output into a .txt file. ...
3 votes
1 answer
301 views

AWS Lambda function to update newly added DynamoDB records

Approach All data is submitted into my DynamoDB from another Lambda > API Integration function whereas the lastUpdated row gets inserted as ...
  • 45
1 vote
0 answers
70 views

Jenkins Powershell script to copy code from Github to AWS Lambda

This is a <10 line Jenkins powershell build step triggered by a git commit which zips code and copies to lambda. Seems to work fine for javascript projects. I check for the zip file and then remove ...
3 votes
3 answers
351 views

Upload .JSON product list to MongoDB and upload image to AWS S3

I'm using Node/Express/Mongoose to accept a JSON file containing a list of product details. These products are looped through, the images are uploaded to AWS S3, and the product is either accepted or ...
  • 165
3 votes
1 answer
3k views

Elastic Beanstalk configuration using ebextensions container_commands combining leader_only and test

For our Django app running on AWS Elastic Beanstalk, we use a web-server/worker setup. The same code is deployed to the web-server and worker environments, and environment-specific configuration is ...
  • 151
5 votes
2 answers
145 views

Simple server log backup script utilising AWS

I have server side log files written to various directories throughout the course of a week. I have the need to optionally compress, and archive these files to AWS. The script I have come up with ...
  • 53
2 votes
1 answer
44 views

More elegant way to construct the S3 path name with given options

I have a method that generates the S3 key prefix according to the provided args. However, my gut feeling is that this method is not as elegant as it should be, maybe I just don't have an eye for it. ...
  • 123
6 votes
1 answer
307 views

Select and delete objects from S3 container

GitLab saves files with the pattern 1530410429_2018_07_01_11.0.1_gitlab_backup.tar (epoch, date, version, fixed string) to an S3 container. I wanted to create a ...
3 votes
0 answers
1k views

AWS Lambda Function to Register in Cognito

This is the entire code for a Lambda function that registers a new user in Amazon Cognito. The email, username, and password are passed to the Lambda function. The client ID is stored in an ...
  • 461
0 votes
1 answer
336 views

Validation of AWS tags

I have to validate Tags with Rules for AWS resource. Input contains list of dictionaries. ...
3 votes
3 answers
812 views

Optimizing Node.js http get for AWS Lambda

I have the following Node.JS (runtime v6.10) executing on AWS Lambda: ...
  • 131
3 votes
1 answer
120 views

Creating a presigned URL

I've created software in Go to create a presigned URL: ...
1 vote
0 answers
428 views

Go program to generate S3 presigned PUT URL

I'm have a made a simple Go program to generate a s3 presigned put url: ...
  • 185
4 votes
0 answers
2k views

AWS EC2 metadata fetcher in Python

Update https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html This seems to be the schema for the instance metadata, though the way they describe it isn't really handy for ...
  • 141
2 votes
1 answer
367 views

node.js connecting to s3 and youtube

I have a task to connect to S3 get a JSON file with the names of artists, use the youtube-data-api and search for the top 10 songs. Then send it back in a given ...
  • 5,311
1 vote
1 answer
1k views

AWS Lambda Python function, to enable / disable access to a VPC environment

We have a requirement that access to an AWS environment is enabled on demand, and said access disabled afterwards. In each case (up / down), this means we will do two things: Enable access Turn on ...
  • 416
5 votes
2 answers
119 views

Bash Database Backup to AWS S3

I've written a bash script that backs up databases and uploads the backups to an S3 bucket. I have the bucket configured to delete backups that are older than 90 days. Is there anything that can be ...
  • 175
3 votes
2 answers
177 views

Authenticating to Amazon EC2 Container Registry

New to bash. The script I wrote works, but I don't believe it is written in the best way possible. The script only assesses how to execute the aws ecr get-login ...
4 votes
2 answers
134 views

Sign an EC2/S3 URL Version 3

From the previous 2 versions: Sign an EC2/S3 URL Version 2 I fixed the comments from the previous version. Most of the code remains the same (so I have split out the new code for review). The main ...
  • 92.9k
2 votes
1 answer
335 views

Checking a DB table before doing an insert using AWS Lambda

Pretty new to AWS Lambda and Javascript, and particularly Promises. Simple check db table before insert. It all seems to be working - I'm just wondering if there was a neater way of doing any of ...
  • 57
2 votes
1 answer
64 views

Sign an EC2/S3 URL Version 2

Follow up on previous review: Sign an EC2/S3 URL I have fixed the previous comments. I have also added the code needed to sign HTTP header fields as part of the request. ...
  • 92.9k
4 votes
2 answers
193 views

Sign an EC2/S3 URL

The other day I found myself trying to walk through the instructions for signing an S3 URL. http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html Not as trivial as the ...
  • 92.9k
4 votes
1 answer
1k views

Implement a boto3 RDS generator in Python

Boto3 does not seem to implement a generator for RDS instances, instead offering a marker and pagination feature. This code does work for my use case and I have been using it for the entire day ...
  • 153