Questions tagged [amazon-web-services]
Amazon Web Services (AWS) is a collection of commercial services centered around virtualized, commodity and clustered computing, networking, database and storage, as well as supporting tools that are offered through infrastructure as a service.
68
questions
4
votes
2
answers
126
views
Reading S3 files from two different buckets but adding it to one list variable
I have a below code which reads all the S3 files from a particular S3 bucket and then it adds all those files in a ...
- 43
0
votes
0
answers
43
views
Copy AWS S3 objects into DB
I have written a code which fetched the S3 objects from AWS s3 using S3 sdk and stores the same in our DB, the only problem is the task is repeated for three different services, the only thing is ...
- 11
0
votes
0
answers
74
views
Go efficiently retrieve and process large volume of AWS s3 objects
Like the title says, code below tries to efficiently grab and handle a large number (1m+) of objects from an aws s3 bucket. Right now on a 10gb lambda function it can handle ~2m objects in ~10s
End ...
- 1,037
2
votes
1
answer
59
views
Bash script to take user input to set AWS envrionment variable the first time and automatically in subsequent calls
Background
Our team has a utility called envmgr to set aws iam roles. The credentials are only good for 4 hours or so. As such, I have a number of scripts that call the command ...
- 311
2
votes
1
answer
163
views
Read bunch of files in parallel and populate concurrent map
I am trying to read bunch of S3 parquet files in parallel from a S3 bucket. After reading all these files, I am populating my products and ...
- 219
5
votes
1
answer
236
views
Rust CLI tool to synchronize files to S3
Learning Rust, I've written a small CLI tool which will
Fetch the existing data in S3
Iterate through local folders
Upload the files which don't exist in S3
I'd be keen to hear anything I might have ...
- 11.6k
7
votes
2
answers
472
views
A Hit Counter For AWS Python Lambda powered with DynamoDB
I'm learning how to use DynamoDB and various AWS services, so I decided to write a simple hit counter application for my education of how to use DynamoDB. This application deduplicates hits from the ...
- 875
1
vote
1
answer
74
views
Archiving MongoDB data to S3, with retryable steps
I have a python script to archive some data (export from mongo to a local json file and then upload it to s3)
Is there a better way to implement/improve the current flow? Each step should be be ...
- 203
3
votes
1
answer
207
views
Delete older EBS Volume snapshots except few with certain tags using Python, Lambda
I am learning Python and managed to accomplish the above. But my script doesn't seem optimal. Could someone help to take a review and suggest for betterment?
...
2
votes
1
answer
57
views
PHP filesystem on a shared file system
I have a project which we need to migrate from an on-prem solution to the AWS cloud. Currently, the software is running on a single instance. It's working fine but the business want the software to be ...
- 23
1
vote
1
answer
50
views
Cleaning Netflix-style recommendation engine datasets
I am working on a portfolio project that deals with creating a Netflix-style recommendation engine for movies. This code is currently ran locally but I will need to upload it into Jupyter Notebooks ...
- 11
1
vote
0
answers
318
views
Golang DB connection with AWS SSM Parameters and context
I'm working on setting up an app backend that's deployed automatically using infrastructure-as-code and a ci/cd pipeline. As a result of that I need the app server to gather DB credentials ...
- 1,037
3
votes
0
answers
73
views
Bash script for rebalancing weighted round robin
I have written a tool that is used to rebalance weights in a weighted round robin in AWS. I am quite happy with it whereas I would welcome feedback from any Bash programmers.
...
- 305
4
votes
0
answers
2k
views
Copying multiple S3 files in parallel using aws shell command
Any improvements to this welcome, I have got it working and I'm happy with it. I'm not really all that proficient with bash shell scripts though.
Problem:
AWS can copy multiple files pretty much as ...
- 143
4
votes
1
answer
59
views
scrape AWS ECS and store summary in S3
This is my scraper for AWS ECS (Elastic Container Service). My primary use for this is to grep through the resulting file to see which cluster a given service is ...
- 2,847
5
votes
1
answer
520
views
Designing a Simple Class that caches files from AWS
Background
I have a pipeline that I run and it spins up multiple EC2 instances that process many jobs. I noticed that in many parts of my pipeline i am downloading files from amazon s3 multiple times. ...
- 387
2
votes
0
answers
28
views
Code and Inputs on to use AWS temporary Credentials
I am adding below code for review as i am bit new to boto3. This Script adds tags (key and value) to All Volumes of EC2 listed in Excel. Is there any more effective way i could have written this? Also ...
- 21
3
votes
1
answer
62
views
Python AWS Lambda, why did updating in place instead of creating and appending to a new list not decrease memory usage?
I have a lambda function that ingests a CSV, reads it as a list of dicts, modifies them (updating the Keys, adding a few values) and submits them to a firehose. Initially I had the following code:
<...
- 31
2
votes
1
answer
107
views
Send Data from DynamoDB to Lambda (C#) and to Azure Queue
I am a beginner and trying to send data from AWS DynamoDB to Azure Queues. Note that this code will be invoked 10,000 and alot more. Can you guys review it once.
...
- 607
3
votes
0
answers
110
views
SageMakerTrainingOperator in Airflow
I have made an operator (surrounded by others operators) for training a model in sagemaker in airflow and I have doubts how would it be more readable or more pythonic. First I did this:
...
3
votes
1
answer
2k
views
Exporting data from PostgreSQL as CSV to S3 bucket
I have a general question regarding DRY. As you can see there are several parts such as io.StringIO(), csv.writer, ...
- 241
1
vote
0
answers
97
views
AWS plugin to create a CloudFormation stack if it doesn't exist
The following Python class is what I am trying to refactor in the case where I have to add validation of the existence of an AWS cloudformation stack before creating it in the method create_cfn_stack :...
- 163
1
vote
1
answer
57
views
Enumerating AWS instances in Rails, with error handling
I have created a method, which is raising the same exception in case of any error:
...
- 119
1
vote
0
answers
72
views
AWS Lambda stack definition YAML
Design
My desired outcome is as follows:
S3 event triggers Lambda Function startTextractStateMachine
Lambda Function ...
- 151
3
votes
0
answers
261
views
User registration in Actix w/ DynamoDB
I am trying to puzzle out api based user registration for a service I am building, but I have a strong feeling that my code is not optimal.
It feels like passing "state" in the way I did prevents the ...
- 113
1
vote
1
answer
399
views
Python functions to save JSON and H5 to local files and S3
I'm trying to copy files to S3 after writing them locally:
I have 2 functions which I need to do this. What is the best way to refactor the code that exists in both.
...
- 601
3
votes
0
answers
95
views
Validating Amazon CloudFormation templates using Rspec
I have created an open source project that uses Rspec to validate the formatting of AWS::CloudFormation::Init YAML blocks in CloudFormation templates.1
My project is here.
Some general concerns:
Is ...
- 305
4
votes
1
answer
703
views
Running AWS commands works ok, but the code seems messy and I don't like the time.sleep bit
This is what I've come up with for a class that handles parsing AWS Textract output. So far it only gets the AWS Textract output into a .txt file.
...
- 151
3
votes
1
answer
301
views
AWS Lambda function to update newly added DynamoDB records
Approach
All data is submitted into my DynamoDB from another Lambda > API Integration function whereas the lastUpdated row gets inserted as ...
- 45
1
vote
0
answers
70
views
Jenkins Powershell script to copy code from Github to AWS Lambda
This is a <10 line Jenkins powershell build step triggered by a git commit which zips code and copies to lambda. Seems to work fine for javascript projects. I check for the zip file and then remove ...
- 19
3
votes
3
answers
351
views
Upload .JSON product list to MongoDB and upload image to AWS S3
I'm using Node/Express/Mongoose to accept a JSON file containing a list of product details. These products are looped through, the images are uploaded to AWS S3, and the product is either accepted or ...
- 165
3
votes
1
answer
3k
views
Elastic Beanstalk configuration using ebextensions container_commands combining leader_only and test
For our Django app running on AWS Elastic Beanstalk, we use a web-server/worker setup. The same code is deployed to the web-server and worker environments, and environment-specific configuration is ...
- 151
5
votes
2
answers
145
views
Simple server log backup script utilising AWS
I have server side log files written to various directories throughout the course of a week. I have the need to optionally compress, and archive these files to AWS.
The script I have come up with ...
- 53
2
votes
1
answer
44
views
More elegant way to construct the S3 path name with given options
I have a method that generates the S3 key prefix according to the provided args. However, my gut feeling is that this method is not as elegant as it should be, maybe I just don't have an eye for it. ...
- 123
6
votes
1
answer
307
views
Select and delete objects from S3 container
GitLab saves files with the pattern 1530410429_2018_07_01_11.0.1_gitlab_backup.tar (epoch, date, version, fixed string) to an S3 container.
I wanted to create a ...
- 63
3
votes
0
answers
1k
views
AWS Lambda Function to Register in Cognito
This is the entire code for a Lambda function that registers a new user in Amazon Cognito.
The email, username, and password are passed to the Lambda function. The client ID is stored in an ...
- 461
0
votes
1
answer
336
views
Validation of AWS tags
I have to validate Tags with Rules for AWS resource.
Input contains list of dictionaries. ...
- 123
3
votes
3
answers
812
views
Optimizing Node.js http get for AWS Lambda
I have the following Node.JS (runtime v6.10) executing on AWS Lambda:
...
- 131
3
votes
1
answer
120
views
1
vote
0
answers
428
views
Go program to generate S3 presigned PUT URL
I'm have a made a simple Go program to generate a s3 presigned put url:
...
- 185
4
votes
0
answers
2k
views
AWS EC2 metadata fetcher in Python
Update
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html
This seems to be the schema for the instance metadata, though the way they describe it isn't really handy for ...
- 141
2
votes
1
answer
367
views
node.js connecting to s3 and youtube
I have a task to connect to S3 get a JSON file with the names of artists, use the youtube-data-api and search for the top 10 songs.
Then send it back in a given ...
- 5,311
1
vote
1
answer
1k
views
AWS Lambda Python function, to enable / disable access to a VPC environment
We have a requirement that access to an AWS environment is enabled on demand, and said access disabled afterwards. In each case (up / down), this means we will do two things:
Enable access
Turn on ...
- 416
5
votes
2
answers
119
views
Bash Database Backup to AWS S3
I've written a bash script that backs up databases and uploads the backups to an S3 bucket. I have the bucket configured to delete backups that are older than 90 days.
Is there anything that can be ...
- 175
3
votes
2
answers
177
views
Authenticating to Amazon EC2 Container Registry
New to bash. The script I wrote works, but I don't believe it is written in the best way possible.
The script only assesses how to execute the aws ecr get-login ...
- 212
4
votes
2
answers
134
views
Sign an EC2/S3 URL Version 3
From the previous 2 versions:
Sign an EC2/S3 URL Version 2
I fixed the comments from the previous version.
Most of the code remains the same (so I have split out the new code for review). The main ...
- 92.9k
2
votes
1
answer
335
views
Checking a DB table before doing an insert using AWS Lambda
Pretty new to AWS Lambda and Javascript, and particularly Promises.
Simple check db table before insert.
It all seems to be working - I'm just wondering if there was a neater way of doing any of ...
- 57
2
votes
1
answer
64
views
Sign an EC2/S3 URL Version 2
Follow up on previous review:
Sign an EC2/S3 URL
I have fixed the previous comments.
I have also added the code needed to sign HTTP header fields as part of the request.
...
- 92.9k
4
votes
2
answers
193
views
Sign an EC2/S3 URL
The other day I found myself trying to walk through the instructions for signing an S3 URL.
http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html
Not as trivial as the ...
- 92.9k
4
votes
1
answer
1k
views
Implement a boto3 RDS generator in Python
Boto3 does not seem to implement a generator for RDS instances, instead offering a marker and pagination feature. This code does work for my use case and I have been using it for the entire day ...
- 153