Questions tagged [amazon-web-services]

Amazon Web Services (AWS) is a collection of commercial services centered around virtualized, commodity and clustered computing, networking, database and storage, as well as supporting tools that are offered through infrastructure as a service.

Filter by
Sorted by
Tagged with
3
votes
1answer
27 views

scrape AWS ECS and store summary in S3

This is my scraper for AWS ECS (Elastic Container Service). My primary use for this is to grep through the resulting file to see which cluster a given service is ...
4
votes
1answer
475 views

Designing a Simple Class that caches files from AWS

Background I have a pipeline that I run and it spins up multiple EC2 instances that process many jobs. I noticed that in many parts of my pipeline i am downloading files from amazon s3 multiple times. ...
2
votes
0answers
16 views

Code and Inputs on to use AWS temporary Credentials

I am adding below code for review as i am bit new to boto3. This Script adds tags (key and value) to All Volumes of EC2 listed in Excel. Is there any more effective way i could have written this? Also ...
3
votes
1answer
33 views

Python AWS Lambda, why did updating in place instead of creating and appending to a new list not decrease memory usage?

I have a lambda function that ingests a CSV, reads it as a list of dicts, modifies them (updating the Keys, adding a few values) and submits them to a firehose. Initially I had the following code: <...
2
votes
1answer
31 views

Send Data from DynamoDB to Lambda (C#) and to Azure Queue

I am a beginner and trying to send data from AWS DynamoDB to Azure Queues. Note that this code will be invoked 10,000 and alot more. Can you guys review it once. ...
1
vote
0answers
56 views

Getting around CORS error for displaying images on my website

I have my images stored in an S3 bucket and I am serving them from AWS CloudFront. I have created 2 CNAME records for the CloudFront: mywebsite.com and ...
3
votes
0answers
48 views

SageMakerTrainingOperator in Airflow

I have made an operator (surrounded by others operators) for training a model in sagemaker in airflow and I have doubts how would it be more readable or more pythonic. First I did this: ...
2
votes
1answer
416 views

Exporting data from PostgreSQL as CSV to S3 bucket

I have a general question regarding DRY. As you can see there are several parts such as io.StringIO(), csv.writer, ...
1
vote
0answers
75 views

AWS plugin to create a CloudFormation stack if it doesn't exist

The following Python class is what I am trying to refactor in the case where I have to add validation of the existence of an AWS cloudformation stack before creating it in the method create_cfn_stack :...
1
vote
1answer
42 views

Enumerating AWS instances in Rails, with error handling

I have created a method, which is raising the same exception in case of any error: ...
1
vote
0answers
45 views

AWS Lambda stack definition YAML

Design My desired outcome is as follows: S3 event triggers Lambda Function startTextractStateMachine Lambda Function ...
3
votes
0answers
114 views

User registration in Actix w/ DynamoDB

I am trying to puzzle out api based user registration for a service I am building, but I have a strong feeling that my code is not optimal. It feels like passing "state" in the way I did prevents the ...
1
vote
1answer
156 views

Python functions to save JSON and H5 to local files and S3

I'm trying to copy files to S3 after writing them locally: I have 2 functions which I need to do this. What is the best way to refactor the code that exists in both. ...
2
votes
0answers
57 views

Validating Amazon CloudFormation templates using Rspec

I have created an open source project that uses Rspec to validate the formatting of AWS::CloudFormation::Init YAML blocks in CloudFormation templates.1 My project is here. Some general concerns: Is ...
4
votes
1answer
286 views

Running AWS commands works ok, but the code seems messy and I don't like the time.sleep bit

This is what I've come up with for a class that handles parsing AWS Textract output. So far it only gets the AWS Textract output into a .txt file. ...
3
votes
1answer
112 views

AWS Lambda function to update newly added DynamoDB records

Approach All data is submitted into my DynamoDB from another Lambda > API Integration function whereas the lastUpdated row gets inserted as ...
1
vote
0answers
62 views

Jenkins Powershell script to copy code from Github to AWS Lambda

This is a <10 line Jenkins powershell build step triggered by a git commit which zips code and copies to lambda. Seems to work fine for javascript projects. I check for the zip file and then remove ...
3
votes
3answers
175 views

Upload .JSON product list to MongoDB and upload image to AWS S3

I'm using Node/Express/Mongoose to accept a JSON file containing a list of product details. These products are looped through, the images are uploaded to AWS S3, and the product is either accepted or ...
2
votes
0answers
1k views

Elastic Beanstalk configuration using ebextensions container_commands combining leader_only and test

For our Django app running on AWS Elastic Beanstalk, we use a web-server/worker setup. The same code is deployed to the web-server and worker environments, and environment-specific configuration is ...
4
votes
1answer
82 views

Simple server log backup script utilising AWS

I have server side log files written to various directories throughout the course of a week. I have the need to optionally compress, and archive these files to AWS. The script I have come up with ...
2
votes
1answer
40 views

More elegant way to construct the S3 path name with given options

I have a method that generates the S3 key prefix according to the provided args. However, my gut feeling is that this method is not as elegant as it should be, maybe I just don't have an eye for it. ...
6
votes
1answer
148 views

Select and delete objects from S3 container

GitLab saves files with the pattern 1530410429_2018_07_01_11.0.1_gitlab_backup.tar (epoch, date, version, fixed string) to an S3 container. I wanted to create a ...
3
votes
0answers
815 views

AWS Lambda Function to Register in Cognito

This is the entire code for a Lambda function that registers a new user in Amazon Cognito. The email, username, and password are passed to the Lambda function. The client ID is stored in an ...
0
votes
1answer
165 views

Validation of AWS tags

I have to validate Tags with Rules for AWS resource. Input contains list of dictionaries. ...
3
votes
3answers
709 views

Optimizing Node.js http get for AWS Lambda

I have the following Node.JS (runtime v6.10) executing on AWS Lambda: ...
2
votes
1answer
94 views

Creating a presigned URL

I've created software in Go to create a presigned URL: ...
1
vote
0answers
249 views

Go program to generate S3 presigned PUT URL

I'm have a made a simple Go program to generate a s3 presigned put url: ...
4
votes
0answers
959 views

AWS EC2 metadata fetcher in Python

Update https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html This seems to be the schema for the instance metadata, though the way they describe it isn't really handy for ...
2
votes
1answer
310 views

node.js connecting to s3 and youtube

I have a task to connect to S3 get a JSON file with the names of artists, use the youtube-data-api and search for the top 10 songs. Then send it back in a given ...
2
votes
1answer
35 views

S3 Bash Tools Part 3

This is a follow on to a previous question: Part 3: User Scripts All the code I am submitting here is available in github The reason to write these tools is so I can prototype and quickly verify that ...
1
vote
0answers
22 views

S3 Bash Tools Part 2

This is a follow on to a previous question: Part 2: Upload/Download All the code I am submitting here is available in github The reason to write these tools is so I can prototype and quickly verify ...
2
votes
0answers
79 views

S3 Tools in Bash

This is a follow on to a previous question: Part 1: Signature All the code I am submitting here is available in github The reason to write these tools is so I can prototype and quickly verify ...
1
vote
1answer
1k views

AWS Lambda Python function, to enable / disable access to a VPC environment

We have a requirement that access to an AWS environment is enabled on demand, and said access disabled afterwards. In each case (up / down), this means we will do two things: Enable access Turn on ...
4
votes
2answers
93 views

Bash Database Backup to AWS S3

I've written a bash script that backs up databases and uploads the backups to an S3 bucket. I have the bucket configured to delete backups that are older than 90 days. Is there anything that can be ...
3
votes
2answers
142 views

Authenticating to Amazon EC2 Container Registry

New to bash. The script I wrote works, but I don't believe it is written in the best way possible. The script only assesses how to execute the aws ecr get-login ...
4
votes
2answers
111 views

Sign an EC2/S3 URL Version 3

From the previous 2 versions: Sign an EC2/S3 URL Version 2 I fixed the comments from the previous version. Most of the code remains the same (so I have split out the new code for review). The main ...
2
votes
1answer
269 views

Checking a DB table before doing an insert using AWS Lambda

Pretty new to AWS Lambda and Javascript, and particularly Promises. Simple check db table before insert. It all seems to be working - I'm just wondering if there was a neater way of doing any of ...
2
votes
1answer
61 views

Sign an EC2/S3 URL Version 2

Follow up on previous review: Sign an EC2/S3 URL I have fixed the previous comments. I have also added the code needed to sign HTTP header fields as part of the request. ...
4
votes
2answers
186 views

Sign an EC2/S3 URL

The other day I found myself trying to walk through the instructions for signing an S3 URL. http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html Not as trivial as the ...
4
votes
1answer
1k views

Implement a boto3 RDS generator in Python

Boto3 does not seem to implement a generator for RDS instances, instead offering a marker and pagination feature. This code does work for my use case and I have been using it for the entire day ...
3
votes
1answer
369 views

Analyze terraform template files to return infrastructure cost

I am trying to write a program I am calling "terraform cashier." Terraform is a tool released by hashicorp to allow you to codify your infrastructure. I thought it would be useful to know how much ...
2
votes
0answers
163 views

Concurrent AWS S3 copying of images listed in MySQL

I created the following one off script and it did the task quite well (1000s of requests / second with low CPU usage - outperformed the previous python script in every aspect). But I am wondering if ...
4
votes
0answers
308 views

Test cases for Ansible lambda module for deployment of AWS Lambdas

I have written a set of test cases which cover the ansible lambda module. The entire source is available on GitHub, forked from ansible/ansible The original file is visible on github as is the ...
6
votes
1answer
3k views

Java 8 spliterator for paged results

Summary: I am using an API which returns paged results. I want to have these results as Java 8 Stream and implemented a ...
4
votes
0answers
562 views

Deploy a Python script as an AWS Lambda micro web service

I'm still new to python, but my history is in the C/C++ world with plenty of Makefiles, though I've rarely written from scratch such elaborate ones as this. The Hitchhiker's Guide to Python gave me ...
3
votes
0answers
168 views

ECMAScript 6 amazon lambda image processing

I use this script in amazon lambda for image processing. Script should do: download image and watermark from amazon S3 resize image to two sizes put watermark on it and annotate with text put resized ...
3
votes
1answer
61 views

Rewrite Amazon s3 key

I've create a function that rewrites the key or "path" of an object in s3. By default, Amazon Web Services Firehose writes to s3 in the format of ...
2
votes
1answer
2k views

Class of file uploading to S3 using boto with multipart supported

I'm writing an app by Flask with a feature to upload large file to S3 and made a class to handle this. I have some targets in writing code: Code must be easy to understand and maintain. (Yes, ...
1
vote
0answers
130 views

AWS Redshift wrapper class that automates similar types of loads from S3

I have a AWS Redshift wrapper class that automates similar types of loads from S3 for me, and I have recently adapted it to work for Spark jobs, which don't require a manifest, and instead need a ...
1
vote
1answer
2k views

Node.js put records to Kinesis with infinite retry strategy.

My application is really easy. It's going to add 500 records at a time to AWS Kinesis and if there's an error occur I'll just add that back to the queue and it's going to be retry. It's going well but ...