Next Page: 10000
|Cache||Network Capture, a Python Module
Sat, 01/19/2019 - 09:52 PM
TCPdump is one of my favorite network packet analysis tools in existence today. In the past year I have enjoyed TCPdump so much that I have wanted to learn more about the project. So, I thought a good place to start would be to get involved contributing on GitHub. My contributions are not much, but they help me learn the ins and outs of the project and how to use some of the more advanced features as I come up in issues or pull requests. During my time thus far I have noticed that a lot of filtering questions tend to come up on how to use filters or how to interpret the documentation for using filtering with TCPdump. Either way, there is a demand for TCPdump users to analyze traffic and pick out only the valuable pieces out from a search criteria. This make sense, but it also made me think... what if there was an easy way to just plug in search criteria based on a network interface or host address and have TCPdump write that information out to a text file. That way it was easy to quickly dissect and if users wanted to do a deeper dive then there was a pcap file for analysis as well. This sounded to me like an opportunity, and that's where the idea for Network Capture, a Python Module was born from.
Note: Network Capture is only available as of right now for Linux. macOS and Windows are still being developed. Network Capture does use Python 3 as it does utilize asyncio.
Using Network Capture
Network Capture is a thin wrapper around TCPdump that allows users to plug in TCPdump arguments along with a comma separated list of keywords to search on. Network Capture then uses Python to execute a packet capture and perform a text based filter based upon your input. The packet capture is written to a standard pcap file and the text based filter is written to a txt file when a keyword match is found. The ASCII output is also written to STDOUT in your terminal, so you can see the info from the traffic being captured and a message will be displayed when a filtering keyword is found in the capture.
Below is an example where a capture is executed on the interface en0 and a text based search is performed for the keywords: "error,get,ssl,https,http." I ran the search for a few seconds and was able to capture 2 results referencing a CNAME with SSL in the capture info.
# Capture all ASCII output containing error,get,ssl,https,http from the interface en0 $ python3 network_capture.py -i en0 -keys error,get,ssl,https,http # In under 2 seconds I captured this traffic to my txt file. (IP's are hidden for security purposes) ** Keyword found. Writing to log ** 09:12:23.718569 IP *.*.*.* > *.*.*.*: 21122 2/0/0 CNAME gstaticadssl.l.google.com., A 184.108.40.206 (87) ** Keyword found. Writing to log ** 09:12:24.405874 IP *.*.*.* > *.*.*.*: 22411 5/0/0 CNAME nagano-4814.herokussl.com., CNAME elb052915-208107455.us-east-1.elb.amazonaws.com., A *.*.*.*, A *.*.*.*, A *.*.*.* (180)
Network Module is still very much a work in progress but the initial functionality should be available to test out on any Linux system with TCPdump installed. As mentioned above, macOS and Windows support is still being tested but I would love to hear if anyone tries Network Capture on either of these platforms. I have 3 main goals for the immediate future of the project:
Please check out the project on my GitHub here, and if you have any questions, comments, or concerns, I'd love to hear about them so please leave a comment and I will get back to you as soon as possible. Thank you!
Add new comment
Just about every platform and programming language has way more features than any one person can explore. That’s a real shame because a lot of these overlooked features are really cool. Thankfully for Objective-C and Cocoa/UIKit we have today’s wonderful iOS resource, NSHipster.
NSHipster - A journal of the overlooked bits in Obj-C and Cocoa
NSHipster is the brainchild of Mattt Thompson (yes, that’s Mattt with 3 T’s…he’s the Mobile Lead and Heroku and the creator of the wonderful AFNetworking library) and is a weekly journal of some of the cooler but often overlooked aspects of Objective-C and Cocoa/UIKit. The oldest article listed is dated July 12, 2012 so there are plenty of articles to read here.
One of the more recent articles dealt with iOS7. With Apple touting 1500 new APIs there was a ton of new material in iOS7. Rather than hitting on the ones everyone else was talking about, NSHipster chose to focus on some of the more interesting bits that might go overlooked like CIDetectorSmile and CIDetectorEyeBlink, AVSpeechSynthesizer, and MKDistanceFormatter.
Other posts are centered around improving your productivity as a developer. If you’re an Xcode developer, you’ll definitely want to check out the posts about Xcode key bindings and gestures and Xcode snippets. The post about xctool is all about helping you control the build process for your applications.
If books are more your thing, you’ll definitely want to check out the first edition of NSHipster as a PDF. The book combines articles from the site as well as new material written just for the book. It’s available for $19.
NSHipster is an amazing resource that looks into the uncharted bits and pieces of Objective-C and Cocoa/UIKit. Since many of the APIs and features discussed on the site are applicable to Xamarin developers, people doing iOS development using C# should definitely check this out as well. There are some great articles to read on NSHipster and I’d recommend reading them all!
Today, we’re thrilled to announce four new trusted data integrations that allow data to flow seamlessly and securely between Heroku and external resources in public clouds and private data centers:
These integrations expand Heroku's security and trust boundary to cover the connections to external resources and the data that passes through them. They enable true multi-cloud app and data architectures and keep developers focused on delivering value versus managing infrastructure. Data is the driving force in modern app development, and these integrations further enhance its value on Heroku by exposing new options for enrichment, analysis, learning, archiving, and more.
Customers are increasingly working with sensitive and regulated data on Heroku and other public clouds or in private data centers. Looking across their use cases, workflows, and challenges, we see two requests emerge:
The use of sensitive and regulated data enables more personalized apps and unique experiences. Working with sensitive and regulated data also introduces greater legal complexities, especially when data crosses cloud boundaries. Heroku’s trusted and compliant data services minimize this risk, so organizations can stay focused on innovating with their data.
First, Heroku Shield provides a set of Heroku platform services that offer additional security features needed for building and running sensitive and regulated data applications. Next, Shield versions of Heroku Postgres, Heroku Redis, and Apache Kafka on Heroku are dedicated, network-isolated data services with strict security rules and compliance standards. And now, our new family of trusted data integrations allows Heroku managed data services to connect to and exchange data with other public clouds or private data centers.
All new trusted data integrations are enabled as of today, included at no additional charge, durable across maintenances and HA failovers, and available in all six Private and Shield Spaces global regions: Sydney, Tokyo, Frankfurt, Dublin, Oregon, and Virginia. Read on for more information on what’s new and how to get started.
This integration allows customers to easily encrypt and mutually authenticate connections between Private and Shield Postgres databases and resources running in other public clouds and private data centers.
Heroku Postgres via mutual TLS requires that both the server and the client verify their certificates and identities to ensure that each one is authenticated and authorized to share data. For additional security, Heroku requires a whitelisted IP or IP range for the client and valid Heroku Postgres credentials. We also log the creation of a mutual TLS connection, notify admin members on the account, and periodically send reminder notifications as long as it is live.
The entire mutual TLS configuration and lifecycle is managed by Heroku to maintain security and meet compliance standards. It’s designed to be configured once and updated every year with new certificates, so the integration recedes into the background of the developer workflow. Get started with Heroku Postgres via mutual TLS.
Earlier this year, we released Heroku Postgres via PrivateLink , which enabled Heroku Postgres databases in Private Spaces to integrate with resources in one or more Amazon VPCs. PrivateLink connections are secure and stable by default because traffic stays on the AWS private network; once a PrivateLink is set up, there is no brittle networking configuration to manage.
We now provide PrivateLink support for Heroku Postgres in Shield Spaces, so that sensitive and regulated data can flow securely and seamlessly between Heroku and AWS. We now log the creation of a PrivateLink, notify admin members on the account, and periodically send reminder notifications as long as it is live. We have also applied these changes to the Private Space version. Get started with Heroku Postgres via PrivateLink.
We also now provide the same PrivateLink support for Apache Kafka on Heroku in Private and Shield Spaces. Just over a month ago, we released Apache Kafka on Heroku Shield and it too now has the ability to integrate with Amazon VPCs for true multi-cloud architectures and best-of-breed solutions. We log, notify, and remind customers as long as this integration is live. Get started with Apache Kafka on Heroku via PrivateLink.
Finally, we now provide the same PrivateLink support for Heroku Redis in Private Spaces . Likewise, we log, notify, and remind customers while the integration is live. Get started with Heroku Redis via PrivateLink.
Heroku balances developer agility and flexibility with enterprise safety and security. Our new trusted data integrations enable sensitive and regulated data to be used across multiple clouds. This allows for true multi-cloud app and data architectures that integrate resources from Heroku, public clouds, and private data centers.
We built these new Trusted data integrations for you and we’re excited to see what you build with them. Please send any feedback our way.
As an experiment to see how static typing could help improve our team’s Ruby experience, we introduced Sorbet into a greenfield codebase with a team of 4 developers. Our theory was that adding static type checking through Sorbet could help us catch bugs before they go into production, make refactoring easier, and improve the design of our code. The short answer is that yes, it did all of that! Read on to learn a little more about what it was like to build in a type safe Ruby.
Ruby is a dynamic language, which means that types are checked when the code is run. If you try to call a method on an object that does not exist, the compiler will not complain, you'll only find out about that error when the code is executed and you get a
Static languages avoid this problem. In a static language, such as Java, the compiler is told or can interpret the type of each variable, and return values from a method or function. This means, among other things, that if you try to call a method that does not exist on an object you will get an error at compile time.
Statically typed languages usually have more overhead in that you need to declare the types of your objects. The added advantage is that they can possibly prevent errors in your code before they're exposed to a user.
Which is where Sorbet comes in. Originally developed by the Stripe team, it lets you use Ruby like you normally would, but also gain the advantages of static type checking. Sorbet accomplishes this in two ways, the first is by using static linting of the files. While that can help you enforce types to a certain extent, the more effective way of type checking is to have the linting happen at runtime. Sorbet will inject itself into each method call and verify that the parameters and return value match what is defined in your signature. This checking adds practically no additional processing time, so there’s no performance issues to worry about.
In the Ruby example above, instead of getting a
Sorbet is designed to be added to existing codebases so the development team has spent a lot of time perfecting that experience. The process of getting the project set up on basic type checking was very easy, you only need to run
For code that is not part of your application, Sorbet provides the option to use RBI files, which are similar to header files in other languages. They are files that allow you to add the method signatures of code that you do not control. This feature is useful to make sure that when you’re using code from a gem, you can still get the benefits of static typing.
Instead of making you do the work of defining a new RBI file for each gem you use, Sorbet also has a GitHub repo where users can add RBI files for gems and version them, this is similar to the TypeScript DefinitleyTyped repo. They provide a command line tool that will scan your gems and pull down any RBI files that are available. These files are then committed to your repo. Gem authors can also provide an RBI file in the gem source, as Sorbet supports that directly, but most authors have not added signatures.
The meta-programming features of Ruby often come in direct conflict with the static typing of Sorbet. One of these issues is Rails and ActiveRecord. When using an ActiveRecord model, you will have methods defined on the class, based on columns in the database. If you would like to be able to use static typing when interacting with these models, you must create RBI files for your models and add the signatures for the methods as you use them.
There is a gem, sorbet-rails, that will attempt to analyze your Rails app and generate the RBI files automatically. As with Sorbet, the project is still young. It was helpful in jump starting some of our typing, but it still misses or mistypes certain fields.
Sadly, very little editor integration is supported at the moment. The Sorbet team is still keeping their plugins closed source until they’re ready for release. For editors that support a way to easily add custom checks, such as Ale in Vim, the CLI works great and is fast enough to give quick feedback. I’m sure that the development experience will improve significantly once the editor integration is available.
We had a strong belief that Sorbet could improve our development experience by lowering our errors and showing us possible problems earlier in our local development. We were very pleased to find that as soon as we added Sorbet and types to our project, Sorbet immediately pointed out a problem in our code! As we continued to add types, Sorbet exposed more issues. I've noted a few of the more interesting problems Sorbet helped us to resolve.
Sorbet complained that we were not properly checking the return type of a method that we called. We did not check that because the same method was called previously in the method and the assumption would be that the underlying value would not change. While that is probably true in almost every case, given our knowledge of that class, we can avoid the problem by assigning the variable out. In this case, Sorbet made our code more reliable.
We decided that we would try our best to not allow nils into our type signatures unless it was unavoidable. This meant that we would rewrite our code where necessary to accommodate this decision. Early into this process, Sorbet made it apparent that we had some columns in our database that we had allowed to be null when that was not what we wanted. This prompted a migration on our database to prevent future problems.
Using Sorbet to define the params and return type can sometimes provide a small benefit by removing tests. In particular there were some tests in our app that were ensuring that the methods returned
Time will tell if Sorbet is a good long term addition, but in the short term, our experience has been very positive. Getting the initial project up and running was very quick and most of the bugs we hit were related to the young age of the Sorbet project. The project maintainers have built a system that stays out of the way until you want it to help and when it does, it provides a solid experience.
Check out the in-browser Sorbet Playground to try Sorbet out without having to download and set up the gem in your codebase.
Next Page: 10000
© Googlier LLC, 2019