Questions tagged [amazon-web-services]

Amazon Web Services (AWS) is a collection of commercial services centered around virtualized, commodity and clustered computing, networking, database and storage, as well as supporting tools that are offered through infrastructure as a service.

Filter by
Sorted by
Tagged with
2
votes
1answer
63 views

Exporting data from PostgreSQL as CSV to S3 bucket

I have a general question regarding DRY. As you can see there are several parts such as io.StringIO(), csv.writer, ...
1
vote
0answers
48 views

AWS plugin to create a CloudFormation stack if it doesn't exist

The following Python class is what I am trying to refactor in the case where I have to add validation of the existence of an AWS cloudformation stack before creating it in the method create_cfn_stack :...
0
votes
1answer
25 views

Enumerating AWS instances in Rails, with error handling

I have created a method, which is raising the same exception in case of any error: ...
1
vote
0answers
25 views

AWS Lambda stack definition YAML

Design My desired outcome is as follows: S3 event triggers Lambda Function startTextractStateMachine Lambda Function ...
3
votes
0answers
64 views

User registration in Actix w/ DynamoDB

I am trying to puzzle out api based user registration for a service I am building, but I have a strong feeling that my code is not optimal. It feels like passing "state" in the way I did prevents the ...
1
vote
1answer
59 views

Python functions to save JSON and H5 to local files and S3

I'm trying to copy files to S3 after writing them locally: I have 2 functions which I need to do this. What is the best way to refactor the code that exists in both. ...
2
votes
0answers
40 views

Validating Amazon CloudFormation templates using Rspec

I have created an open source project that uses Rspec to validate the formatting of AWS::CloudFormation::Init YAML blocks in CloudFormation templates.1 My project is here. Some general concerns: Is ...
4
votes
1answer
156 views

Running AWS commands works ok, but the code seems messy and I don't like the time.sleep bit

This is what I've come up with for a class that handles parsing AWS Textract output. So far it only gets the AWS Textract output into a .txt file. ...
3
votes
1answer
94 views

AWS Lambda function to update newly added DynamoDB records

Approach All data is submitted into my DynamoDB from another Lambda > API Integration function whereas the lastUpdated row gets inserted as ...
0
votes
0answers
45 views

Boto3 Python 2.7 class for encrypting Amazon images

I would appreciate a review of this Python code I have written, particularly with respect to general coding style, the design, and how I have organised the code as classes. Other feedback most welcome ...
1
vote
0answers
49 views

Jenkins Powershell script to copy code from Github to AWS Lambda

This is a <10 line Jenkins powershell build step triggered by a git commit which zips code and copies to lambda. Seems to work fine for javascript projects. I check for the zip file and then remove ...
3
votes
3answers
146 views

Upload .JSON product list to MongoDB and upload image to AWS S3

I'm using Node/Express/Mongoose to accept a JSON file containing a list of product details. These products are looped through, the images are uploaded to AWS S3, and the product is either accepted or ...
2
votes
0answers
470 views

Elastic Beanstalk configuration using ebextensions container_commands combining leader_only and test

For our Django app running on AWS Elastic Beanstalk, we use a web-server/worker setup. The same code is deployed to the web-server and worker environments, and environment-specific configuration is ...
4
votes
1answer
65 views

Simple server log backup script utilising AWS

I have server side log files written to various directories throughout the course of a week. I have the need to optionally compress, and archive these files to AWS. The script I have come up with ...
2
votes
1answer
38 views

More elegant way to construct the S3 path name with given options

I have a method that generates the S3 key prefix according to the provided args. However, my gut feeling is that this method is not as elegant as it should be, maybe I just don't have an eye for it. ...
6
votes
1answer
107 views

Select and delete objects from S3 container

GitLab saves files with the pattern 1530410429_2018_07_01_11.0.1_gitlab_backup.tar (epoch, date, version, fixed string) to an S3 container. I wanted to create a ...
3
votes
0answers
685 views

AWS Lambda Function to Register in Cognito

This is the entire code for a Lambda function that registers a new user in Amazon Cognito. The email, username, and password are passed to the Lambda function. The client ID is stored in an ...
0
votes
1answer
103 views

Validation of AWS tags

I have to validate Tags with Rules for AWS resource. Input contains list of dictionaries. ...
3
votes
3answers
680 views

Optimizing Node.js http get for AWS Lambda

I have the following Node.JS (runtime v6.10) executing on AWS Lambda: ...
2
votes
1answer
79 views

Creating a presigned URL

I've created software in Go to create a presigned URL: ...
1
vote
0answers
168 views

Go program to generate S3 presigned PUT URL

I'm have a made a simple Go program to generate a s3 presigned put url: ...
4
votes
0answers
731 views

AWS EC2 metadata fetcher in Python

Update https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html This seems to be the schema for the instance metadata, though the way they describe it isn't really handy for ...
2
votes
1answer
264 views

node.js connecting to s3 and youtube

I have a task to connect to S3 get a JSON file with the names of artists, use the youtube-data-api and search for the top 10 songs. Then send it back in a given ...
2
votes
1answer
33 views

S3 Bash Tools Part 3

This is a follow on to a previous question: Part 3: User Scripts All the code I am submitting here is available in github The reason to write these tools is so I can prototype and quickly verify ...
1
vote
0answers
20 views

S3 Bash Tools Part 2

This is a follow on to a previous question: Part 2: Upload/Download All the code I am submitting here is available in github The reason to write these tools is so I can prototype and quickly ...
1
vote
0answers
63 views

S3 Tools in Bash

This is a follow on to a previous question: Part 1: Signature All the code I am submitting here is available in github The reason to write these tools is so I can prototype and quickly verify ...
1
vote
1answer
1k views

AWS Lambda Python function, to enable / disable access to a VPC environment

We have a requirement that access to an AWS environment is enabled on demand, and said access disabled afterwards. In each case (up / down), this means we will do two things: Enable access Turn on ...
4
votes
2answers
83 views

Bash Database Backup to AWS S3

I've written a bash script that backs up databases and uploads the backups to an S3 bucket. I have the bucket configured to delete backups that are older than 90 days. Is there anything that can be ...
3
votes
2answers
109 views

Authenticating to Amazon EC2 Container Registry

New to bash. The script I wrote works, but I don't believe it is written in the best way possible. The script only assesses how to execute the aws ecr get-login ...
4
votes
2answers
102 views

Sign an EC2/S3 URL Version 3

From the previous 2 versions: Sign an EC2/S3 URL Version 2 I fixed the comments from the previous version. Most of the code remains the same (so I have split out the new code for review). The main ...
2
votes
1answer
183 views

Checking a DB table before doing an insert using AWS Lambda

Pretty new to AWS Lambda and Javascript, and particularly Promises. Simple check db table before insert. It all seems to be working - I'm just wondering if there was a neater way of doing any of ...
2
votes
1answer
60 views

Sign an EC2/S3 URL Version 2

Follow up on previous review: Sign an EC2/S3 URL I have fixed the previous comments. I have also added the code needed to sign HTTP header fields as part of the request. ...
4
votes
2answers
171 views

Sign an EC2/S3 URL

The other day I found myself trying to walk through the instructions for signing an S3 URL. http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html Not as trivial as the ...
4
votes
1answer
900 views

Implement a boto3 RDS generator in Python

Boto3 does not seem to implement a generator for RDS instances, instead offering a marker and pagination feature. This code does work for my use case and I have been using it for the entire day ...
3
votes
1answer
344 views

Analyze terraform template files to return infrastructure cost

I am trying to write a program I am calling "terraform cashier." Terraform is a tool released by hashicorp to allow you to codify your infrastructure. I thought it would be useful to know how much ...
2
votes
0answers
158 views

Concurrent AWS S3 copying of images listed in MySQL

I created the following one off script and it did the task quite well (1000s of requests / second with low CPU usage - outperformed the previous python script in every aspect). But I am wondering if ...
4
votes
0answers
278 views

Test cases for Ansible lambda module for deployment of AWS Lambdas

I have written a set of test cases which cover the ansible lambda module. The entire source is available on GitHub, forked from ansible/ansible The original file is visible on github as is the ...
6
votes
1answer
2k views

Java 8 spliterator for paged results

Summary: I am using an API which returns paged results. I want to have these results as Java 8 Stream and implemented a ...
4
votes
0answers
534 views

Deploy a Python script as an AWS Lambda micro web service

I'm still new to python, but my history is in the C/C++ world with plenty of Makefiles, though I've rarely written from scratch such elaborate ones as this. The Hitchhiker's Guide to Python gave me ...
3
votes
0answers
165 views

ECMAScript 6 amazon lambda image processing

I use this script in amazon lambda for image processing. Script should do: download image and watermark from amazon S3 resize image to two sizes put watermark on it and annotate with text put resized ...
3
votes
1answer
58 views

Rewrite Amazon s3 key

I've create a function that rewrites the key or "path" of an object in s3. By default, Amazon Web Services Firehose writes to s3 in the format of ...
2
votes
1answer
2k views

Class of file uploading to S3 using boto with multipart supported

I'm writing an app by Flask with a feature to upload large file to S3 and made a class to handle this. I have some targets in writing code: Code must be easy to understand and maintain. (Yes, ...
1
vote
0answers
114 views

AWS Redshift wrapper class that automates similar types of loads from S3

I have a AWS Redshift wrapper class that automates similar types of loads from S3 for me, and I have recently adapted it to work for Spark jobs, which don't require a manifest, and instead need a ...
1
vote
1answer
1k views

Node.js put records to Kinesis with infinite retry strategy.

My application is really easy. It's going to add 500 records at a time to AWS Kinesis and if there's an error occur I'll just add that back to the queue and it's going to be retry. It's going well but ...
5
votes
2answers
114 views

Rails controller to manage EC2 instances

I did a small sample application for an Interview. Since I didn't got accepted I would like your opinion on how to improve it. The aux methods are private. I have a ...
4
votes
1answer
68 views

Simple shell script to backup to Amazon over iSCSI

The script works but I know that it can be improved. Please note that my scripting skills are very basic :) (as it can be seen in my Frankenstein script – I took bits and pieces of the code from ...
1
vote
0answers
607 views

Unit tests for a Redshift wrapper class

I'm somewhat new to TDD and unit testing, but I've written a suite of unit tests to check my functionality. The basic classes automate simple load and unload operations from s3 and redshift, and ...
2
votes
1answer
1k views

Redshift Load and Unload Wrapper class

We do constant traffic with our Redshift tables, and so I created a wrapper class that will allow for custom sql to be ran (or a default generic stmt), and can run a ...
2
votes
1answer
2k views
2
votes
1answer
2k views

Listing EC2 instances

I've got this massive nested for loop with a few if and else statements in nodeJs. The output of all of this is just string which is combined from different values out of JSon object. What I want to ...