Questions tagged [amazon-web-services]

Amazon Web Services (AWS) is a collection of commercial services centered around virtualized, commodity and clustered computing, networking, database and storage, as well as supporting tools that are offered through infrastructure as a service.

Filter by
Sorted by
Tagged with
2 votes
2 answers
235 views

Distributed lock system using AWS S3

I need a distributed lock system for a service I'm writing. (There can be multiple instances of this service running at a time, and I need to make sure that only one instance does a particular cron ...
threed's user avatar
  • 161
4 votes
2 answers
150 views

Reading S3 files from two different buckets but adding it to one list variable

I have a below code which reads all the S3 files from a particular S3 bucket and then it adds all those files in a ...
rosed's user avatar
  • 43
0 votes
0 answers
49 views

Copy AWS S3 objects into DB

I have written a code which fetched the S3 objects from AWS s3 using S3 sdk and stores the same in our DB, the only problem is the task is repeated for three different services, the only thing is ...
jack's user avatar
  • 11
2 votes
1 answer
130 views

Bash script to take user input to set AWS envrionment variable the first time and automatically in subsequent calls

Background Our team has a utility called envmgr to set aws iam roles. The credentials are only good for 4 hours or so. As such, I have a number of scripts that call the command ...
Hofbr's user avatar
  • 321
2 votes
1 answer
306 views

Read bunch of files in parallel and populate concurrent map

I am trying to read bunch of S3 parquet files in parallel from a S3 bucket. After reading all these files, I am populating my products and ...
AndyP's user avatar
  • 229
5 votes
1 answer
302 views

Rust CLI tool to synchronize files to S3

Learning Rust, I've written a small CLI tool which will Fetch the existing data in S3 Iterate through local folders Upload the files which don't exist in S3 I'd be keen to hear anything I might have ...
Jeroen Vannevel's user avatar
7 votes
2 answers
528 views

A Hit Counter For AWS Python Lambda powered with DynamoDB

I'm learning how to use DynamoDB and various AWS services, so I decided to write a simple hit counter application for my education of how to use DynamoDB. This application deduplicates hits from the ...
HSchmale's user avatar
  • 875
1 vote
1 answer
99 views

Archiving MongoDB data to S3, with retryable steps

I have a python script to archive some data (export from mongo to a local json file and then upload it to s3) Is there a better way to implement/improve the current flow? Each step should be be ...
KeykoYume's user avatar
  • 203
3 votes
1 answer
336 views

Delete older EBS Volume snapshots except few with certain tags using Python, Lambda

I am learning Python and managed to accomplish the above. But my script doesn't seem optimal. Could someone help to take a review and suggest for betterment? ...
Vidarbha Yalagadde's user avatar
2 votes
1 answer
82 views

PHP filesystem on a shared file system

I have a project which we need to migrate from an on-prem solution to the AWS cloud. Currently, the software is running on a single instance. It's working fine but the business want the software to be ...
spectrum's user avatar
1 vote
1 answer
51 views

Cleaning Netflix-style recommendation engine datasets

I am working on a portfolio project that deals with creating a Netflix-style recommendation engine for movies. This code is currently ran locally but I will need to upload it into Jupyter Notebooks ...
cloud_user's user avatar
1 vote
0 answers
367 views

Golang DB connection with AWS SSM Parameters and context

I'm working on setting up an app backend that's deployed automatically using infrastructure-as-code and a ci/cd pipeline. As a result of that I need the app server to gather DB credentials ...
Coupcoup's user avatar
  • 1,067
3 votes
0 answers
86 views

Bash script for rebalancing weighted round robin

I have written a tool that is used to rebalance weights in a weighted round robin in AWS. I am quite happy with it whereas I would welcome feedback from any Bash programmers. ...
Alex Harvey's user avatar
4 votes
0 answers
3k views

Copying multiple S3 files in parallel using aws shell command

Any improvements to this welcome, I have got it working and I'm happy with it. I'm not really all that proficient with bash shell scripts though. Problem: AWS can copy multiple files pretty much as ...
PhilHibbs's user avatar
  • 143
4 votes
1 answer
64 views

scrape AWS ECS and store summary in S3

This is my scraper for AWS ECS (Elastic Container Service). My primary use for this is to grep through the resulting file to see which cluster a given service is ...
chicks's user avatar
  • 2,845
5 votes
1 answer
552 views

Designing a Simple Class that caches files from AWS

Background I have a pipeline that I run and it spins up multiple EC2 instances that process many jobs. I noticed that in many parts of my pipeline i am downloading files from amazon s3 multiple times. ...
Dinero's user avatar
  • 387
2 votes
0 answers
35 views

Code and Inputs on to use AWS temporary Credentials

I am adding below code for review as i am bit new to boto3. This Script adds tags (key and value) to All Volumes of EC2 listed in Excel. Is there any more effective way i could have written this? Also ...
saurabh's user avatar
  • 21
3 votes
1 answer
66 views

Python AWS Lambda, why did updating in place instead of creating and appending to a new list not decrease memory usage?

I have a lambda function that ingests a CSV, reads it as a list of dicts, modifies them (updating the Keys, adding a few values) and submits them to a firehose. Initially I had the following code: <...
guestAWSdev's user avatar
2 votes
1 answer
117 views

Send Data from DynamoDB to Lambda (C#) and to Azure Queue

I am a beginner and trying to send data from AWS DynamoDB to Azure Queues. Note that this code will be invoked 10,000 and alot more. Can you guys review it once. ...
Unbreakable's user avatar
3 votes
0 answers
120 views

SageMakerTrainingOperator in Airflow

I have made an operator (surrounded by others operators) for training a model in sagemaker in airflow and I have doubts how would it be more readable or more pythonic. First I did this: ...
Javier Lopez Tomas's user avatar
3 votes
1 answer
2k views

Exporting data from PostgreSQL as CSV to S3 bucket

I have a general question regarding DRY. As you can see there are several parts such as io.StringIO(), csv.writer, ...
Joey Coder's user avatar
1 vote
0 answers
104 views

AWS plugin to create a CloudFormation stack if it doesn't exist

The following Python class is what I am trying to refactor in the case where I have to add validation of the existence of an AWS cloudformation stack before creating it in the method create_cfn_stack :...
S.K. Venkat's user avatar
1 vote
1 answer
62 views

Enumerating AWS instances in Rails, with error handling

I have created a method, which is raising the same exception in case of any error: ...
Rajkaran Mishra's user avatar
1 vote
0 answers
72 views

AWS Lambda stack definition YAML

Design My desired outcome is as follows: S3 event triggers Lambda Function startTextractStateMachine Lambda Function ...
ChumiestBucket's user avatar
3 votes
0 answers
285 views

User registration in Actix w/ DynamoDB

I am trying to puzzle out api based user registration for a service I am building, but I have a strong feeling that my code is not optimal. It feels like passing "state" in the way I did prevents the ...
Ethan McCue's user avatar
1 vote
1 answer
444 views

Python functions to save JSON and H5 to local files and S3

I'm trying to copy files to S3 after writing them locally: I have 2 functions which I need to do this. What is the best way to refactor the code that exists in both. ...
gogasca's user avatar
  • 601
3 votes
0 answers
99 views

Validating Amazon CloudFormation templates using Rspec

I have created an open source project that uses Rspec to validate the formatting of AWS::CloudFormation::Init YAML blocks in CloudFormation templates.1 My project is here. Some general concerns: Is ...
Alex Harvey's user avatar
4 votes
1 answer
809 views

Running AWS commands works ok, but the code seems messy and I don't like the time.sleep bit

This is what I've come up with for a class that handles parsing AWS Textract output. So far it only gets the AWS Textract output into a .txt file. ...
ChumiestBucket's user avatar
3 votes
1 answer
320 views

AWS Lambda function to update newly added DynamoDB records

Approach All data is submitted into my DynamoDB from another Lambda > API Integration function whereas the lastUpdated row gets inserted as ...
Curtis's user avatar
  • 35
3 votes
3 answers
376 views

Upload .JSON product list to MongoDB and upload image to AWS S3

I'm using Node/Express/Mongoose to accept a JSON file containing a list of product details. These products are looped through, the images are uploaded to AWS S3, and the product is either accepted or ...
DasBeasto's user avatar
  • 155
3 votes
1 answer
3k views

Elastic Beanstalk configuration using ebextensions container_commands combining leader_only and test

For our Django app running on AWS Elastic Beanstalk, we use a web-server/worker setup. The same code is deployed to the web-server and worker environments, and environment-specific configuration is ...
djvg's user avatar
  • 161
5 votes
2 answers
151 views

Simple server log backup script utilising AWS

I have server side log files written to various directories throughout the course of a week. I have the need to optionally compress, and archive these files to AWS. The script I have come up with ...
awaugh's user avatar
  • 53
2 votes
1 answer
53 views

More elegant way to construct the S3 path name with given options

I have a method that generates the S3 key prefix according to the provided args. However, my gut feeling is that this method is not as elegant as it should be, maybe I just don't have an eye for it. ...
Infamous's user avatar
  • 123
6 votes
1 answer
317 views

Select and delete objects from S3 container

GitLab saves files with the pattern 1530410429_2018_07_01_11.0.1_gitlab_backup.tar (epoch, date, version, fixed string) to an S3 container. I wanted to create a ...
asylumine's user avatar
3 votes
0 answers
1k views

AWS Lambda Function to Register in Cognito

This is the entire code for a Lambda function that registers a new user in Amazon Cognito. The email, username, and password are passed to the Lambda function. The client ID is stored in an ...
Lou Bagel's user avatar
  • 461
0 votes
1 answer
430 views

Validation of AWS tags

I have to validate Tags with Rules for AWS resource. Input contains list of dictionaries. ...
Vivek Sable's user avatar
3 votes
3 answers
835 views

Optimizing Node.js http get for AWS Lambda

I have the following Node.JS (runtime v6.10) executing on AWS Lambda: ...
gvasquez's user avatar
  • 131
3 votes
1 answer
122 views

Creating a presigned URL

I've created software in Go to create a presigned URL: ...
920smmsmsms's user avatar
1 vote
0 answers
429 views

Go program to generate S3 presigned PUT URL

I'm have a made a simple Go program to generate a s3 presigned put url: ...
FXux's user avatar
  • 185
4 votes
0 answers
2k views

AWS EC2 metadata fetcher in Python

Update https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html This seems to be the schema for the instance metadata, though the way they describe it isn't really handy for ...
nodakai's user avatar
  • 141
2 votes
1 answer
368 views

node.js connecting to s3 and youtube

I have a task to connect to S3 get a JSON file with the names of artists, use the youtube-data-api and search for the top 10 songs. Then send it back in a given ...
Gilad's user avatar
  • 5,321
1 vote
1 answer
1k views

AWS Lambda Python function, to enable / disable access to a VPC environment

We have a requirement that access to an AWS environment is enabled on demand, and said access disabled afterwards. In each case (up / down), this means we will do two things: Enable access Turn on ...
kafka's user avatar
  • 416
5 votes
2 answers
138 views

Bash Database Backup to AWS S3

I've written a bash script that backs up databases and uploads the backups to an S3 bucket. I have the bucket configured to delete backups that are older than 90 days. Is there anything that can be ...
Enijar's user avatar
  • 175
3 votes
2 answers
181 views

Authenticating to Amazon EC2 Container Registry

New to bash. The script I wrote works, but I don't believe it is written in the best way possible. The script only assesses how to execute the aws ecr get-login ...
TheAuzzieJesus's user avatar
4 votes
2 answers
140 views

Sign an EC2/S3 URL Version 3

From the previous 2 versions: Sign an EC2/S3 URL Version 2 I fixed the comments from the previous version. Most of the code remains the same (so I have split out the new code for review). The main ...
Martin York's user avatar
  • 94.3k
2 votes
1 answer
339 views

Checking a DB table before doing an insert using AWS Lambda

Pretty new to AWS Lambda and Javascript, and particularly Promises. Simple check db table before insert. It all seems to be working - I'm just wondering if there was a neater way of doing any of ...
L G's user avatar
  • 57
2 votes
1 answer
117 views

Sign an EC2/S3 URL Version 2

Follow up on previous review: Sign an EC2/S3 URL I have fixed the previous comments. I have also added the code needed to sign HTTP header fields as part of the request. ...
Martin York's user avatar
  • 94.3k
4 votes
2 answers
198 views

Sign an EC2/S3 URL

The other day I found myself trying to walk through the instructions for signing an S3 URL. http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html Not as trivial as the ...
Martin York's user avatar
  • 94.3k
4 votes
1 answer
1k views

Implement a boto3 RDS generator in Python

Boto3 does not seem to implement a generator for RDS instances, instead offering a marker and pagination feature. This code does work for my use case and I have been using it for the entire day ...
Purefan's user avatar
  • 153
4 votes
1 answer
414 views

Analyze terraform template files to return infrastructure cost

I am trying to write a program I am calling "terraform cashier." Terraform is a tool released by hashicorp to allow you to codify your infrastructure. I thought it would be useful to know how much ...
Bjorn248's user avatar