buffuniq is a simple command-line utility written in Go that filters out duplicate lines from standard input, printing only the first occurrence of each line to standard output. It uses a circular buffer to keep track of unique lines up to a specified size.
buffuniq [-n SIZE]-n SIZE: Specifies the size of the circular buffer used to store unique lines. If no size is provided, the default size is 100.
To filter out duplicate lines from a file named input.txt and print only unique lines:
cat input.txt | buffuniq -n 100- Ensure you have Go installed on your system. You can download it from https://golang.org/dl/.
- Clone this repository or download the source code.
- Navigate to the directory containing the source code.
- Build the binary:
go build -o buffuniq buffuniq.go
- Move the
buffuniqbinary to a directory in your PATH, for example/usr/local/bin.
The program reads lines from standard input and uses a circular buffer (-n SIZE) to keep track of unique lines. When a line is read, it checks if the line has been seen before using the IsNew method. If the line is new, it prints the line; otherwise, it skips it.
The program uses a slice (uniqBuff) to implement the circular buffer. The bufI index keeps track of the current position in the buffer, and size defines its capacity. When the buffer is full and a new line is added, the oldest entry is replaced by the new line.
Feel free to fork this repository and submit pull requests for bug fixes or enhancements. Any contributions are welcome!
This project is licensed under the MIT License - see the LICENSE file for details.
If you have any questions or feedback, please open an issue on GitHub or contact me directly.
For more detailed information and additional examples, refer to the source code comments in buffuniq.go.