Splunk ES CI/CD pipeline
Splunk Cloud CI/CD pipeline - GitOps
Dev stage
Workflow
When the configurations get pushed to a feature branch or to the dev branch, the Jenkins pipeline get triggered. This would send a notification to the developer in the MS Teams channel acknowledging that the build process has been started.
The pipeline will execute the stages relevant to dev changeset. this includes running validations and generating changelogs.
The generated reports get send to the developer via MS Teams channel.
The changelog report includes the details about the changes that would take effect on Splunk staging environment (When changes get committed to staging branch).
When you push configurations to feature/dev branch Pipeline runs to execute terraform plan, but nothing is applied to any environment.
(e.g: If the developer places/removes the Splunk app he’s developed from the Splunkapps/deployer-apps directory and pushes the changes in to a feature branch it will switch in to the relevant terraform workspace and grab the environment information which, the changes should be applied. In this case the changes would be tested against the deployment server. Since the changes has been pushed into a feature branch it will perform terraform plan and send the planned changes information back to the developer.)
Test stage
Workflow
When the changes get committed to the test branch, the pipeline would run Splunk docker container and load Eventgen, SimData and Splunk attack data based on the configurations (Please refer Splunk data configuration, Splunk use cases sections for more details) and run test cases and generate reports.
The generated reports at each stage get uploaded to s3 in a folder specific to the user who made the changes and a pre-signed url is generated for each report and send via MS Teams channel. This includes slim validation report, AppInspect report and use-cases report.
Only the apps which have been passed the AppInspection process would get deployed to the test instance.
The apps which have been already deployed (through pipeline) to the Splunk Cloud will get skipped the validation process.
Staging stage
Workflow
Private app installation/uninstallation
• Check if the app contains app manifest file and generate it if it doesn’t contain a manifest file.
• Check if the app has been packaged or not and package the app if it hasn’t been packaged (generate tar.gz).
• Check if the app is valid or not
• The valid packaged apps get pushed to the target Splunk instance
• A list of apps which should deployed on the target instance get created and uploaded to S3
• The packaged apps get uploaded to s3
• The apps get uploaded to target by using ecs exec (ecs execute-command --region <region> --cluster <cluster> --task <task> --container <container> --command "aws s3 cp s3://bucket-name/appname /home/apps/appname --recursive" --interactive)
• Terraform reads the relevant S3 object (apps list) based on the target Splunk instance
• Based on the changes, the target Splunk instance and the environment will be identified
Prod stage
Workflow
When the changes get pushed to the prod branch the pipeline would run the checks to validate that the configurations have been tested on the searchHead in Splunk staging environment. Only the configurations that has been tested on the searchHead in Splunk staging environment will get deployed to Splunk cloud.
To manage private apps and indexes in Splunk cloud, the pipeline utilizes the Splunk ACS API. To manage the other configurations, the pipeline utilizes Splunk terraform provider.
Due to the asynchronous app installation/index creation process, it might take a few minutes for the app installation/index creation process to complete.
Staging environment
Note: Only the changes tested on the searchHead in staging environment will get pushed to Splunk cloud.
Install/Remove Private Apps
Create, install and manage private apps on Splunk instance.
For latest resource argument reference: https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTapps#apps.2Flocal
Steps
Navigate to the Splunk-apps directory and open the searchHead directory.
To create a new splunk app, upload the required app package. The app package needs to be inside a directory. (e.g: searchHead/appfolder/apppackage).
Push the changes to the dev branch and you will receive notification/report related to the changes that will be applied to the staging environment.
To test the changes, push the code to the test branch. This would run the app validations/Cloud vetting and configured use cases against the app (Please refer Splunk data configuration, Splunk use cases sections for more details).
Push the changes to the dev branch and you will receive notification/report related to the changes that will be applied to the staging environment.
To test the changes, push the code to the test branch.
Once succeeded you can push the changes to the staging branch so the changes could get reflected on Splunk staging environment. Since the changes have been made to the searchHead, the changes will get applied to the searchHead of the staging environment.
The changes then can be pushed to the preprod branch to generate a report of what changes would apply to the Splunk cloud upon pushing the changes to prod branch.
Once the changes get validated, the changes can be pushed to the prod branch to apply the changes to the Splunk cloud.
Splunk Data Configuration
Configure Splunk attack data and Splunk SimData to be used to run tests.
SimData
SimData is a tool that generates event data from a simulation of a user-defined scenario. Instead of using a sample set of data that is repetitive and unrealistic, SimData allows you to generate a rich and robust set of events from real-world situations by mimicking how multiple systems work together and affect the performance of your system.
Suppose your system includes a web server and a database server. SimData simulations let you emulate possible outcomes when the database server is impacted by situations such as web traffic, then you can see how overall performance is affected. SimData can correlate the behavior of the web server and the database server so that when the database slows, the web server is also impacted.
Use SimData for:
App development, using SimData to generate data with a specific format to test the different features of your app. SimData allows you to represent scenarios using your data to ensure that those patterns appear in your extractions.
Live demonstrations, using SimData to send events in real time so that you can demonstrate plausible behavior. You can also change variables at runtime to demonstrate how an app responds to different behaviors. The result is a realistic demo, in real time.
The purposes of SimData include:
Using a simple description language with powerful primitives.
Allowing for simple and complex interactions between simulated entities.
Enabling flexible modeling capabilities.
Modeling each entity in a typical simulation as a specific object.
Emulating existing formats for log data.
Using messages to between entities for communication, layered on the Akka message paradigm.
How Simdata works
SimData models your environment as follows:
Systems are referred to as entities.
A bot is a running instance of an entity.
Bots use messages to communicate with each other.
Actions are triggered for conditions that you specify.
Events are emitted from the simulation. An event usually corresponds to a log entry.
For example, for a set of users interacting with a web server, which communicates to a backend database server, the interaction flow might look like this:
The entities are the user, the database server, and the web server. The simulation can model the interaction between multiple users, multiple databases, and multiple web servers. In this example, the model would have one entity for users ("User"), one entity for web servers ("Webserver"), and one entity for database servers ("Database"). The simulation would have one bot for each User that is modeled, and for each WebServer and each Database in the modeled network.
The messages are the requests and responses between the bots in the simulation. As the simulation runs, you could simulate what happens when you flood a web server with traffic by increasing the round-trip time for each request, which triggers an action such as a forced restart of the database. You would model this by updating the state on the bot that represents the overloaded database.
The messages in the diagram are all internal to the simulation and are not visible outside of the simulation. You can then add events from the WebServer and Database entity definitions so that you get logs that reflect the internal states of the bots. These events can be weblog entries from the web server, database logs from the database, and CPU load metrics that reflect the modeled load on the underlying servers.
Eventgen
The Splunk Event Generator (Eventgen) helps you build real-time event generators.
Splunk attack data
A Repository of curated datasets from various attacks to:
Easily develop detections without having to build an environment from scratch or simulate an attack.
Test detections, specifically Splunks Security Content
Replay into streaming pipelines for validating your detections in your production SIEM
Splunk attack data configuration
Steps
Navigate to the Splunk-data-configs directory and open the config.json file in the attackdataConfigs directory.
To add new data set to replay, add a new stanza to the configuration file. To remove data set, remove the required configuration stanza of that data set from the configuration file.
Push the changes to the dev branch and you will receive notification/report related to the changes that will be applied to the staging environment.
To test the changes, push the changes to the test branch. This would run load the configured attack dataset in to Splunk test instance and replay the attack.
This configuration supports the following arguments:
technique_id - (Required) MITTRE technique id.
name - (Required) name of data set to replay.
path - (Required) relative path of raw file.
replay_parameters - (Required) splunk parameters to pass.
enabled - (Required) attack replay enabled indicator.
{
"datasets": [
{
"technique_id": "T1003.002",
"name": "T1003.002_windows_security",
"path": "datasets/attack_techniques/T1003.002/atomic_red_team/windows-security.log",
"replay_parameters": {
"source": "WinEventLog:Security",
"sourcetype": "WinEventLog",
"index": "main"
},
"enabled": true
}
]
}
Splunk SimData configuration
Steps
Navigate to the Splunk-data-configs directory and create a Simulation file in the simdataConfigs directory.
Simulation file (Required): defines the entities being modeled, messages passed between the actors in the system (called 'bots'), and events to be written to a transport such as SplunkHEC.
Navigate to the Splunk-data-configs directory and create a Scene file in the simdataConfigs directory.
Scene file (Required): configures the runtime configuration. This defines where events are written, how many bots are in the simulation, and how messages are connected between entities.
Here's a simple simulation file (hello.simulation) that defines a single type of bot (entity) called Greeter, with an event type called Greeting.
event Greeting {
required: [text];
}
entity Greeter {
every(5s) {
emit Greeting {
text: "Hello, World!";
}
}
}
The simulation requires a corresponding scene file (hello.json). To keep things simple, this scene file defines only a single bot, which will output to the console when the simulation is run, by choosing the default_transport setting Text.
{
"default_transport": "Text",
"entities": [
{
"count": 1,
"entity_name": "Greeter"
}
],
"time_unit": "Seconds",
"update_interval": 1
}
Navigate to the Splunk-data-configs directory and update the simdataconfig.json file in the simdataConfigs directory.
Push the changes to the dev branch and you will receive notification/report related to the changes that will be applied to the staging environment.
To test the changes, push the code to the test branch.
{
"configs": [
{
"configfiles": ["hello"],
"enabled": true
}
]
}
Splunk use-cases/searches
Run use-cases/searches in Splunk test instance.
Steps
Navigate to the Splunk-usecases directory and open the usecases.json file.
To run a new search, add a new stanza to the configuration file. To modify, update the existing search configuration with the updated parameter values. To stop running a search, remove the required configuration stanza of that search from the configuration file or mark false as the enable parameter.
Push the changes to the dev branch and you will receive notification/report related to the changes that will be applied to the staging environment.
To test the changes, push the code to the test branch.
This configuration supports the following arguments:
searches - (Required) search/searches need to run.
enable - (Required) search/searches enabled indicator.
Create/Modify/Remove HEC token
Create or update HTTP Event Collector input configuration tokens.
For latest resource argument reference: https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTinput#data.2Finputs.2Fhttp
Steps
Navigate to the Splunk-hec-tokens directory and open the hec-tokens.json file inside the searchHead directory.
To create a new HEC token, add a new stanza to the configuration file. To modify, update the existing HEC token configuration with the updated parameter values. To remove a HEC token, remove the required configuration stanza of that HEC token from the configuration file.
Push the changes to the dev branch and you will receive notification/report related to the changes that will be applied to the staging environment.
Push the changes to the test branch. (HEC tokens will not get created in the test environment).
Once succeeded you can push to the changes to the staging branch so the changes could get reflected on Splunk staging environment. Since the changes has been made to the searchHead, the changes will get applied to the searchHead of the staging environment.
The HEC token values will get saved as a secret parameter in AWS SSM parameter store. You will receive the ARNs of the created parameters and the parameter names follows the following naming convention. /splunk-hec-tokens/{username}/{splunk partition}/{token name}
The changes then can be pushed to the preprod branch to generate a report of what changes would apply to the Splunk cloud upon pushing the changes to prod branch.
Once the changes get validated, the changes can be pushed to the prod branch to apply the changes to the Splunk cloud.
This configuration supports the following arguments:
name - (Required) Token name (inputs.conf key)
token - (Required) Token value for sending data to collector/event endpoint
indexes - (Required) Set of indexes allowed for events with this token
source - (Required) Default source for events with this token
sourcetype - (Required) Default source type for events with this token
acl - (Required) The app/user context that is the namespace for the resource
{
"tokens": [
{
"name" : "user",
"indexes" : ["vpc-flow-logs"],
"source" : "new:source",
"sourcetype" : "new:sourcetype",
"acl": {
"owner": "admin",
"sharing": "global",
"read": ["admin"],
"write": ["admin"]
}
},
{
"name" : "user-test",
"indexes" : ["vpc-flow-logs"],
"source" : "new:source",
"sourcetype" : "new:sourcetype",
"acl": {
"owner": "admin",
"sharing": "global",
"read": ["admin"],
"write": ["admin"]
}
}
]
}
Comments
Post a Comment