GitMonitor Github敏感信息扫描工具

GitMonitor Github敏感信息扫描工具

个人隐私保护, 技能树, 黑客工具, 黑客技术

GitMonitor工具介绍

GitMonitor是一个Github系统扫描工具，用于根据规则查找在GitHub上泄漏的敏感信息.。我知道，目前有很多非常好的其他工具可用于查找在Github上泄露的敏感信息，我本人目前仍在使用其中一些工具。但是，我认为它们仍然缺少一些功能，例如：

  • 基于规则的扫描工具。
  • 规则机制使我能够以最灵活的方式编写规则。这些规则使我可以按品牌名称，文件格式和语言过滤信息。以及允许我跳过特定的文件格式和语言（搜索规则）。然后，将匹配规则的存储库克隆到本地，然后再开始基于正则表达式（敏感过滤规则）查找存在于本地的敏感信息。为此，您可以定义与公司品牌名称相关的关键字，与公司项目相关的关键字，电子邮件前缀或规则中的其他任何内容。
  • 该工具可以按计划启动，并具有灵活的报告机制。

这就是为什么我创建了此工具-GitMonitor。GitMonitor使用两组不同的规则来查找所需的内容。搜索规则将搜索可能与您的组织或内部项目相关的存储库，或其他任何与本地匹配的存储库。然后，使用敏感过滤规则检查那些存储库是否存在敏感信息。最后，该工具将通过Slack报告。您可以将此工具与Cronjob一起使用，以创建监视系统，以跟踪与在Github上泄漏的组织相关的敏感信息，并通过Slack接收结果。

GitMonitor Github敏感信息扫描工具
GitMonitor Github敏感信息扫描工具

GitMonitor特色

  • 根据规则搜索存储库（搜索规则）。您可以编写规则来搜索可能与您的公司有关的存储库。符合规则的存储库将被克隆到本地。
  • 使用Regex（敏感筛选规则）搜索克隆存储库中存在的敏感信息，以进行分类。
  • 通过Slack报告。
  • 规则和正则表达式分别定义
  • 用户可以轻松直观地定义规则和正则表达式。

GitMonitor安装要求

Python3，Python3-pip
目前在Ubuntu 18.04上测试成功。

GitMonitor安装必要条件/环境库

python3 -m pip install -r requirements.txt

requirements

requests
gitpython
pyyaml

请确保您已安装Pyyaml版本5x或更高版本

在配置文件（config.ini）中填写所需的信息：

[git]
 user = <username_git>
 pass = <password_git>
 url_code = https://api.github.com/search/code?q={}+in:file&sort=indexed&order=desc
 url_repos = https://api.github.com/search/repositories?q={}+size:>0+is:public&sort=indexed&order=desc
 url_commit = https://api.github.com/search/commits?q={}+is:public&sort=indexed&order=desc
 rpp = 50
 
 [slack]
 webhooks = <full_link_webhooks>
 
 
 [path]
 rule = <path to rule folder>
 source = <path to folder to clone repository>
 log = <filename of log>
 
 [msg]
 start = ====================**********====================
 
         *Start scanning at {}*
         _Clone completed successfully:_
 end = ====================**********====================
 
       *Scanning Done at {}*
       _Detected possible repository:_
 all = ====================**********====================

编写规则(搜索规则)将您的规则放在rules目录中：

 id: Project_X_Matching
 key: X
 language:
   - java
 #filename:
 #  - LICENSE
 #extension:
 #  - py
 #  - md
 ignore:
 #  language:
 #    - php
   filename:
     - LICENSE
   extension:
     - html
     - txt

在libs/regex.py文件中定义正则表达式(敏感过滤规则)
regex.py如下

to_match = [
    {'match_regex': 'password', 'match_type': 'Password'},
    {'match_regex': '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3,}', 'match_type': 'IP'},
    {'match_regex': 'username', 'match_type': 'Username'},
    {'match_regex': '([A-Za-z0-9._%%+-]+[a-zA-Z0-9])@([A-Za-z0-9.-]+[a-zA-Z0-9])\.([A-Za-z]{2,})', 'match_type': 'Email'},
    {'match_regex': '(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16,}', 'match_type': "AWS Client ID"},
    {'match_regex': '(?i)aws(.{0,20})\?[\\\'\\\"][0-9a-zA-Z\/+]{40,}[\\\'\\\"]', 'match_type': "AWS Secret Key"},
    {'match_regex': 'amzn\.mws\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12,}', 'match_type': "AWS MWS key"},
    {'match_regex': '\-\-\-\-\-BEGIN PRIVATE KEY\-\-\-\-\-', 'match_type': "PKCS8"},
    {'match_regex': '\-\-\-\-\-BEGIN RSA PRIVATE KEY\-\-\-\-\-', 'match_type': "RSA"},
    {'match_regex': '\-\-\-\-\-BEGIN OPENSSH PRIVATE KEY\-\-\-\-\-', 'match_type': "SSH"},
    {'match_regex': '\-\-\-\-\-BEGIN PGP PRIVATE KEY BLOCK\-\-\-\-\-', 'match_type': "PGP"},
    {'match_regex': '(?i)(facebook|fb)(.{0,20})?[\\\'\\\"][0-9a-f]{32,}[\\\'\\\"]', 'match_type': "Facebook Secret Key"},
    {'match_regex': '(?i)(facebook|fb)(.{0,20})?[\\\'\\\"][0-9]{13,17}[\\\'\\\"]', 'match_type': "Facebook Client ID"},
    {'match_regex': '[f|F][a|A][c|C][e|E][b|B][o|O][o|O][k|K].*[\\\'|\\\"][0-9a-f]{32,}[\\\'|\\\"]', 'match_type': "Facebook Oauth"},
    {'match_regex': 'EAACEdEose0cBA[0-9A-Za-z]+', 'match_type': "Facebook access token"},
    {'match_regex': '(?i)twitter(.{0,20})?[\\\'\\\"][0-9a-z]{35,44}[\\\'\\\"]', 'match_type': "Twitter Secret Key"},
    {'match_regex': '(?i)twitter(.{0,20})?[\\\'\\\"][0-9a-z]{18,25}[\\\'\\\"]', 'match_type': "Twitter Client ID"},
    {'match_regex': '[t|T][w|W][i|I][t|T][t|T][e|E][r|R].{0,30}[\\\'\\\"\\s][0-9a-zA-Z]{35,44}[\\\'\\\"\\s]', 'match_type': "Twitter Oauth"},
    {'match_regex': '(?i)github(.{0,20})?[\\\'\\\"][0-9a-zA-Z]{35,40}[\\\'\\\"]', 'match_type': "Github"},
    {'match_regex': '(?i)linkedin(.{0,20})?[\\\'\\\"][0-9a-z]{12,}[\\\'\\\"]', 'match_type': "LinkedIn Client ID"},
    {'match_regex': '(?i)linkedin(.{0,20})?[\\\'\\\"][0-9a-z]{16,}[\\\'\\\"]', 'match_type': "LinkedIn Secret Key"},
    {'match_regex': 'xox[baprs]-([0-9a-zA-Z]{10,48})?', 'match_type': "Slack"},
    {'match_regex': '\-\-\-\-\-BEGIN EC PRIVATE KEY\-\-\-\-\-', 'match_type': "EC"},
    {'match_regex': '(?i)api_key(.{0,20})?[\\\'|\\\"][0-9a-zA-Z]{32,45}[\\\'|\\\"]', 'match_type': "Generic API key"},
    {'match_regex': '(?i)secret(.{0,20})?[\\\'|\\\"][0-9a-zA-Z]{32,45}[\\\'|\\\"]', 'match_type': "Generic Secret"},
    {'match_regex': 'AIza[0-9A-Za-z\\\-\_]{35,}', 'match_type': "Google API key"},
    {'match_regex': '(?i)(google|gcp|youtube|drive|yt)(.{0,20})?[\\\'\\\"][AIza[0-9a-z\\\-\_]{35,}][\\\'\\\"]', 'match_type': "Google Cloud Platform API key"},
    {'match_regex': '(?i)(google|gcp|auth)(.{0,20})?[\\\'\\\"][0-9]+-[0-9a-z_]{32,}\.apps\.googleusercontent\.com[\\\'\\\"]', 'match_type': "Google OAuth"},
    {'match_regex': 'ya29\.[0-9A-Za-z\-_]+', 'match_type': "Google OAuth access token"},
    {'match_regex': '(?i)heroku(.{0,20})?[\\\'\\\"][0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}[\\\'\\\"]', 'match_type': "Heroku API key"},
    {'match_regex': '(?i)(mailchimp|mc)(.{0,20})?[\\\'\\\"][0-9a-f]{32,}-us[0-9]{1,2}[\\\'\\\"]', 'match_type': "MailChimp API key"},
    {'match_regex': '(?i)(mailgun|mg)(.{0,20})?[\\\'\\\"][0-9a-z]{32,}[\\\'\\\"]', 'match_type': "Mailgun API key"},
    {'match_regex': '[a-zA-Z]{3,10}:\/\/[^\/\s:@]{3,20}:[^\/\s:@]{3,20}@.{1,100}\/?.?', 'match_type': "Password in URL"},
    {'match_regex': 'access_token\$production\$[0-9a-z]{16,}\$[0-9a-f]{32,}', 'match_type': "PayPal Braintree access token"},
    {'match_regex': 'sk_live_[0-9a-z]{32,}', 'match_type': "Picatic API key"},
    {'match_regex': 'https\:\/\/hooks.slack.com\/services\/T[a-zA-Z0-9_]{8}\/B[a-zA-Z0-9_]{10,}\/[a-zA-Z0-9_]{24,}', 'match_type': "Slack Webhook"},
    {'match_regex': '(?i)stripe(.{0,20})?[\\\'\\\"][sk|rk]_live_[0-9a-zA-Z]{24,}', 'match_type': "Stripe API key"},
    {'match_regex': 'sq0atp-[0-9A-Za-z\-_]{22,}', 'match_type': "Square access token"},
    {'match_regex': 'sq0csp-[0-9A-Za-z\\-_]{43,}', 'match_type': "Square OAuth secret"},
    {'match_regex': '(?i)twilio(.{0,20})?[\\\'\\\"][0-9a-f]{32,}[\\\'\\\"]', 'match_type': "Twilio API key"}
    ]

GitMonitor运行方式

python3 gitmonitor.py

您可以使用Cronjob设置该工具自动运行.

GitMonitor下载地址

①GitHub : https://github.com/ddosi/hack
②雨苁网盘: https://w.ddosi.workers.dev/

项目地址

GitHub github.com/Talkaboutcybersecurity/GitMonitor

Leave a Reply

您的电子邮箱地址不会被公开。 必填项已用 * 标注