Questions tagged [awk]

AWK is an interpreted programming language designed for text processing and typically used as a data extraction and reporting tool.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

UNIX calendar(1) in awk

This is a simple implementation of the calendar(1) utility included in some UNIX systems (all BSDs have it, GNU has not). Here's the manual: ...
1
vote
0answers
39 views

detection the selfish node Using awk File [closed]

I am working on detection the selfish node based on this equation Using awk File : (Node Communication Ratio) NCR= ((Get Route Request-(Get Route Request-Reply Route Request))/Get Route Request)*100 ...
2
votes
0answers
44 views

The Blacklist v2

This script gathers domains listed from sources in this JSON file and compiles them into one text file. Text files that have a similar key under the whitelists ...
3
votes
1answer
56 views

Parse log file and send the result to an API

The task is to get the most recent login time of the day for each user. Then send such a result to an API. The log file (file.log) looks like the following (only ...
2
votes
0answers
30 views

bash / awk workflow operating on multi-column files [closed]

I am working on the improvement of the bash-awk workflow designed for the analysis of data in multiple-colums format. The script uses AWK code (which extract numbers from selected columns of the input....
1
vote
0answers
16 views

Printing of status message during execution of the bash script

As a part of my bash routine I am printing some message in terminal regarding status of the workflow. The message in splited into two parts (part 1: begining of the task, part 2: status of its ...
0
votes
1answer
73 views

Compare script using awk and bash

I try to create compare script using awk and another command and running successful. But I think the script I create it's to long. Is there anyone can shorten my script below ? After plan A ...
2
votes
0answers
43 views

Faster algorithm/language for substring sliding window replacement

I had a sample file with a long string >50M like this. ...
2
votes
1answer
52 views

Bash script wallpaper randomizer

I thought it would be fun to write a simple and small bash script which randomly chooses a wallpaper from a given directory of a set of wallpaper images. The code is as follows. ...
1
vote
1answer
70 views

Parse and clean large log files

I have the code below, which works successfully, and is used to parse, clean log files (very large in size) and output into smaller sized files. Output filename is the first 2 characters of each line. ...
0
votes
1answer
25 views

Split large log file like a dictionary [closed]

I have a large log file in the following format. Do note that this is already sorted alphabetically. I wish to split it into smaller pieces, and with file names reflecting the content of the file (...
1
vote
1answer
54 views

Extracting emails from log files

I have huge number of text files, each several hundred MB in size. Unfortunately, they are not all fully standardized in any one format. Plus there is a lot of legacy in here, and a lot of junk and ...
3
votes
1answer
160 views

An Awk program to find all primes till N

I developed Awk code to print all primes till N where N is given as command line argument. I think I took pretty much all the optimizations into account. Could someone help if it could be optimized ...
3
votes
0answers
34 views

Merging multiple awk commands while retaining functionality

Turned in a project a few days ago, and I needed a way to filter and format values off a file. I decided to go for a bash script since it seemed like a good option, and I came up with the following: <...
3
votes
1answer
325 views

Extract multiple attributes values in XML tags

I have a XML file (test.xml) that can be sum up as follow (I filtered it so it will be more readable): ...
2
votes
0answers
47 views

Bash wrapper for connmanctl with autocompletion

So the idea came from the fact that connmanctl only completes filenames/directories when used non-interactively, whereas when used interactively (executing ...
6
votes
3answers
312 views

Count unique IP address in a date range from Apache log

I use the following command to get specific entries between certain dates from Apache log files. Then I filter out my own address and some bots, then I print out the IP, sort it, get the uniques, ...
4
votes
1answer
76 views

Removing duplicate field entries from sorted csv data

Given the following input (cat i.txt), I want to remove duplicate field entries in each of the first three columns and none of the others. ...
0
votes
2answers
61 views

Select a specific line after matching a pattern

I just recently started using AWK and I'm still learning about it. I have solved the problem I'm about to show but I feel it's not the best solution and I'm trying to find a solution to fit within an ...
4
votes
1answer
79 views

Select second field in text table with shell script

I am writing a bash script to mount openshift service accounts into kubernetes objects. The right secret to use is highlighted in this text: $ oc describe sa sa-...
3
votes
2answers
80 views

Code to get the IP address and format it to run in nmap

For scanning my subnet using my IP address, I need to run the following command nmap -sn 192.168.100.* I want to extend this functionality such that I don't have ...
4
votes
0answers
48 views

Summary data using CSV file

Is there the possibility to improve the code below? I am sure the result I got can be achieved with shorter code. Code ...
7
votes
1answer
952 views

Updating my package version using only a Makefile

So I have written a Makefile that has a release function that can be run like: make release bump=patch and I want it to be fully ...
7
votes
3answers
622 views

Print code-fenced sections of a Markdown document

Original code and demo at this gist. Given a Markdown document like ...
3
votes
1answer
51 views

Counts and average code

Using the columns 4 and 2 will create a report like the output file showed below. My code works fine but I believe it can be done shorter. I have a doubt in the part of the split: ...
3
votes
2answers
80 views

Bash script for managing hashtag notes

I have written a simple script for managing hashtag notes. I am not familiar with bash scripting so I would really appreciate it if someone could give me some pointers about it. ...
4
votes
1answer
44 views

Remove every line that include and follows the second occurrence of a given pattern

From a file, I am trying to remove every line that include and follows the second occurrence of a given pattern in Bash (Mac OSX). Note, that a file may have only one match of the given pattern (and ...
4
votes
1answer
81 views

Try every option for a command and make anagrams out of the successful ones

What I have: I am trying to find what words you can make out of Linux command options like ls -cat or ps -elf. To do that I am ...
2
votes
1answer
140 views

Extract thoughput value from JMeter log file

I use JMeter for load testing and want to report error on CI pipeline when application though put is less than 1/sec. JMeter logs summary of results as - ...
3
votes
1answer
90 views

ECG Bash selection tool

I made the following bash script for extracting a group of ECG signals from ECG files. I would like to know if there is any mistakes and/or weaknesses. I have experienced difficulties in integrating ...
5
votes
1answer
2k views

Python library for awk-like file manipulation

I recently published a library for advanced awk-like file manipulation in Python 3. The code can be found here and here is the documentation. It is also available for download from pip (...
3
votes
1answer
311 views

Parsing GTF file using command-line

I am extracting exons details from a GTF file using command line in Unix like cut, awk, grep or sed. input file.gtf: ...
4
votes
2answers
373 views

Generate SQL UPDATE from Excel CSV file

I must write updates in a database from a flat file (CSV). I want to do that in the shell, with tools such as AWK. ...
4
votes
1answer
196 views

Find and report all occurrences of duplicate lines in a text file

I would like you to take a look at this simple script I wrote that's supposed to emulate the Unix sort [FILE] | uniq -cd command. What's different about my script ...
0
votes
1answer
37 views

Loading a file in vim via piped output of ag

I feel as if my scenario is fairly simple but that I might be overcomplicating it. I am searching through files looking for a special condition, that is where the method ...
2
votes
1answer
661 views

Finding hidden field values in a web page before submitting a login form

In order to submit a login form using wget, I need to find the values for the hidden field that were added to the form recently. Grepping for the names gives the ...
2
votes
1answer
58 views

Listing Docker Images without registry URI

I have a private container registry behind a firewall that will need some Google registry images. To make sure things are updated sanely, I'm writing a mirror script in Bash. I have one line that ...
3
votes
2answers
235 views

Advanced CSV-to-SQLite converter

I just wrote an advanced CSV parser that translates CSV to SQL(SQLite Compatible) statements, which are then inserted into the database. It translates variable names in the CSV to values defined in ...
1
vote
1answer
34 views

Print first element of the last line of files when the second element does not match a pattern

I am relatively new to Bash, regular expression and stuff. In a directory, I need to read the last line of all the files that contain the word "fitness", and list (on one line and separated with ...
6
votes
3answers
244 views

Shell script to count chess game outcomes

I came across this blog post by Adam Drake from around a year ago which is now making the rounds again. I made some improvements to his code, but wish to see if there are additional tweaks that could ...
7
votes
1answer
75 views

Calculating percentages in arbitrary number of columns

Given this sample input: ...
5
votes
2answers
3k views

Checking percentage of free memory using top and awk

The following Awk code was created in order to verify if free memory from the top command is less than 20% of the total. I've noticed that I actually do not need the ...
3
votes
1answer
1k views

Analyse an Nginx access log with awk and produce a report

I'm looking for your opinions on anything that I am doing wrong in the application below, such as best practices, glaringly horrible errors or even just your own personal opinion. Task: Grab an Nginx ...
10
votes
3answers
23k views

Extracting data from text file in bash using awk, grep, head and tail

I've been writing bash script on and off, with pretty good results in terms of getting the job done. However, I'm worried that my script might be very ugly, as I am a beginner. I'm looking for advice ...
4
votes
2answers
100 views

Filter shell script to find lines that contain all specified patterns

I wrote a script that does the following: Run another script on the system Filter the output to find lines that contain ALL of the given patterns Pipe the output to a second script on the system I ...
4
votes
1answer
73 views

Extracting performance statistics from FTP session transcript

I've borrowed and written the following code to output the disconnect time. All works well but I'm curious as to how I could tighten/ shorten the code. If anyone feels like having some fun then I'd ...
9
votes
1answer
889 views

Bash script - automate an upload image + description process on a server

I was tasked at work to improve a system where through a web interface a client, which eventually became us, uploads a bunch of images with descriptions (latter coming from csv files), the images are ...
4
votes
3answers
198 views

Removing NULL / empty fields

just wanted to check with you could this be done better: ...
4
votes
1answer
72 views

Input Columns with Awk

I wrote a small script to manipulate fields of a file based on the input of a user. Is there something I could've done better, or perhaps a better refactor? I do know however that I need a way to ...
6
votes
2answers
204 views

Bash log monitoring

I've made a Bash script to monitor some server log files for certain data and my method probably isn't the most efficient. One section specifically bugs me is that I have to write a newline to the ...