database administrator (DBA)

Copyright TechTarget - All rights reserved https://cyber.law.harvard.edu/rss/rss.html Techtarget Feed Generator en Tue, 12 May 2026 04:41:27 GMT https://searchsqlserver.techtarget.com editor@techtarget.com <section class="section main-article-chapter" data-menu-title="What is a database administrator (DBA)?"> <h2 class="section-title">What is a database administrator (DBA)?</h2> A database administrator (DBA) is the information technician responsible for directing and performing all activities related to maintaining and securing a successful <a href="https://www.techtarget.com/searchdatamanagement/definition/database">database</a> environment. A DBA ensures an organization's databases and related applications operate functionally and efficiently. DBAs frequently design and develop new features and debug issues, while working alongside a diverse group of individuals such as developers, <a href="https://www.techtarget.com/searchdatamanagement/definition/data-architect">data architects</a> and business stakeholders. Therefore, they're expected to be proficient in both technical and business skills. </section> <section class="section main-article-chapter" data-menu-title="Why are DBAs important?"> <h2 class="section-title">Why are DBAs important?</h2> For any organization that uses a database management system (<a href="https://www.techtarget.com/searchdatamanagement/definition/database-management-system">DBMS</a>) for mission-critical workloads, it's important to have one or more database administrators on board to ensure that applications have ongoing, uninterrupted access to data. Most organizations use at least one DBMS, and therefore the need for database administrators is greater today than ever before. The following highlights the importance of database administrators: <ul class="default-list"> <li>DBAs are responsible for understanding and managing the overall database environment. By developing and executing a strategic blueprint to follow when deploying databases within their organization, DBAs are instrumental in the ongoing efficacy of applications that rely on databases for data storage and access.</li> <li>Without the DBA's oversight, application and system outages, downtime and slowdowns will inevitably occur. These kinds of problems result in business outages that can negatively affect revenue, customer experiences and business reputation.</li> <li>DBAs monitor database performance, detect <a href="https://www.techtarget.com/searchnetworking/definition/bottleneck">bottlenecks</a> and optimize queries, configurations and indexes to ensure optimal performance and reliability of databases.</li> <li>DBAs are the subject matter experts for database management systems and all related topics, including DBMS setup and configuration; database design; structured query language (<a href="https://www.techtarget.com/searchdatamanagement/definition/SQL">SQL</a>) coding; data extraction, transformation and loading (<a href="https://www.techtarget.com/searchdatamanagement/definition/Extract-Load-Transform-ELT">ETL</a>); test data management; problem resolution; data security and integrity; <a href="https://www.techtarget.com/searchsecurity/tip/Evaluate-cloud-database-security-controls-best-practices">database security</a>; performance management; optimization; and database backup and recovery.</li> <li>DBAs use security techniques such as access controls, <a href="https://www.techtarget.com/searchsecurity/tip/Data-masking-vs-data-encryption-How-do-they-differ">encryption and data masking</a> to protect data integrity and confidentiality of databases, reducing the risks associated with data breaches and compliance violations.</li> <li>DBAs possess the skills to convert unprocessed data into meaningful business intelligence. They're crucial in helping to glean insightful information from data, which is necessary for the organization's decision-making process.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="What are the roles and responsibilities of a DBA?"> <h2 class="section-title">What are the roles and responsibilities of a DBA?</h2> A DBA assumes many different roles and responsibilities within the IT department involving database systems and applications. The roles and responsibilities of a DBA typically include the following: <ul class="default-list"> <li>Database deployment and management. The DBA keeps databases and applications running up to performance, availability and recoverability standards while handling additional functions. When adopting a new DBMS, the DBA is responsible for designing, setting up and maintaining the database system. Often that includes installing the DBMS and setting up the IT infrastructure to enable applications to access databases. For a <a href="https://www.techtarget.com/searchcloudcomputing/definition/cloud-database">cloud database</a> implementation, the DBA isn't responsible for installation but must orchestrate the proper configuration, access and deployment options for their organization's use of the cloud database.</li> <li>Documentation and training. The DBA must also establish policies and procedures pertaining to the management, security, maintenance and use of the database management system. The DBA group creates training materials and instructs employees and developers on the proper access and usage of the DBMS.</li> <li>Troubleshooting. When problems arise, the DBA is the focal point for resolution, including troubleshooting, <a href="https://www.techtarget.com/searchitoperations/definition/root-cause-analysis">root cause analysis</a>, fine tuning and optimizing the performance of tasks and programs that access the database. The DBA must be capable of performing root cause analysis -- identifying the cause of the problem so it can be resolved. This task requires the ability to locate bottlenecks and points of contention, monitor workload and throughput, review SQL performance and optimization, monitor storage space and fragmentation and view and manage the system and DBMS resource use.</li> <li>Backup and disaster recovery. The DBA is responsible for ensuring that databases and data are backed up appropriately and can be recovered correctly and quickly in the event of failure. The DBA also ensures that databases are protected and secured, enacts measures to maintain the database integrity in terms of data accuracy and makes sure unauthorized users can't access the data.</li> <li>Collaboration and integration. The DBA frequently gets pulled into other projects as a subject matter expert on the database. Because databases are at the center of most modern application development projects, the DBA participates in database integration and use in a variety of IT projects. This exposure to many different technologies and experiences can make the DBA a valuable IT technician not just for database-related issues, but for other technologies as well.</li> <li>Efficient database design. Database administrators work with application developers to ensure accurate and efficient application design for database access. DBA tasks include interfacing SQL with traditional programming languages, selecting the type of SQL to use, using middleware and APIs such as <a href="https://www.techtarget.com/searchapparchitecture/definition/RESTful-API">REST</a>, <a href="https://www.techtarget.com/searchoracle/definition/Open-Database-Connectivity">Open Database Connectivity</a>, <a href="https://www.theserverside.com/definition/Java-Database-Connectivity-JDBC">Java Database Connectivity</a> and SQLJ effectively, defining transactions and determining the appropriate use of frameworks such as Java Enterprise Edition and .NET. DBAs might also be asked to modify or write application code to help development projects.</li> <li>Optimizing database performance. Database performance is the optimization of resource usage to increase throughput and minimize contention, enabling the largest possible workload to be processed. Ensuring efficient performance of the database and applications that access it is a core function of database administration.</li> <li>Staying current. As the central point of contact for information about the company's database management systems, DBAs must keep up to date on the latest versions and capabilities of each DBMS.</li> <li>Using the right tools. Administering a DBMS usually requires the use of other tools than those provided with the DBMS. DBAs must understand the strengths and weaknesses of native DBA tools, develop a strategy for addressing the weaknesses and use tools that improve the performance, availability, administration and recovery of the databases they manage.</li> <li>User access and support. A DBA creates new user accounts and provides access permissions. They also create and manage database reports, visualizations and dashboards to help users easily understand the status of their database.</li> <li>Communication. DBAs must have good communication skills. They must be able to converse and work with application programmers, business end users, IT and business managers, data analysts and other DBAs.</li> </ul> <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/the_many_roles_of_a_dba-f.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/the_many_roles_of_a_dba-f_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/the_many_roles_of_a_dba-f_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/the_many_roles_of_a_dba-f.png 1280w" alt="A description of the functions of different DBA roles." height="319" width="560"> <figcaption> DBAs take on a variety of tasks. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> </section> <section class="section main-article-chapter" data-menu-title="What are the different types of DBAs?"> <h2 class="section-title">What are the different types of DBAs?</h2> Many different types of DBAs exist, the most common type being the general-purpose DBA, who performs all types of administrative and data-related work. However, it isn't uncommon for DBAs to focus on specific problem domains. A DBA, for example, might focus entirely on database design, perhaps broken into logical design and physical design; specialize in building systems; concentrate on maintaining and tuning existing systems; or center their attention on narrow areas of database management and administration. Within larger organizations, DBA responsibilities typically are split into separate types of roles. Beyond general-purpose, the primary roles include system DBA, database architect, database analyst, application DBA, task-oriented DBA, performance analyst, <a href="https://www.techtarget.com/searchdatamanagement/definition/data-warehouse">data warehouse</a> administrator and cloud DBA. <h3>System DBA</h3> This role focuses on technical, rather than business, issues. The system DBA is knowledgeable in the arcane technical details of how the database is installed, configured and modified. Typical tasks center on the physical installation and performance of <a href="https://www.techtarget.com/searchdatamanagement/feature/Evaluating-the-different-types-of-DBMS-products">the DBMS software</a> and can include the following: <ul class="default-list"> <li>Installing new software versions and applying fixes.</li> <li>Setting and tuning system parameters.</li> <li>Tuning the operating system, network and transaction processors to work with the DBMS.</li> <li>Ensuring appropriate storage and memory are available for the DBMS.</li> </ul> System DBAs are rarely involved with the actual database and application set up. They might get involved in application tuning when operating system parameters or complex DBMS parameters need to be altered. <h3>Database architect</h3> The primary responsibility of this role is the design and set up of new databases. The database architect designs new databases and database structures for new and existing applications and is rarely involved in the maintenance and tuning of established databases and applications. Typical tasks include the following: <ul class="default-list"> <li>Modeling logical data.</li> <li>Translating logical data models into a physical database design.</li> <li>Analyzing data access requirements to ensure optimal database design and efficient SQL access.</li> <li>Creating <a href="https://www.techtarget.com/searchdatabackup/Create-your-data-backup-strategy-A-comprehensive-guide">backup and recovery strategies</a> for new databases.</li> </ul> <h3>Database analyst</h3> Sometimes junior DBAs are referred to as database analysts. The database analyst's role might be like that of the database architect. The database analyst designation can be another name for a database administrator. <h3>Application DBA</h3> An application DBA focuses on database design and the ongoing database support and administration for a specific application or subset of applications. The application DBA is likely an expert in writing and debugging complex SQL. They understand the best ways to incorporate database requests into application programs. Application DBAs typically are also responsible for managing and refreshing test data for application development teams. Not every organization has an application DBA on staff. In that case, the general-purpose DBA supports specific applications while also maintaining the organization's database environment. But even with an application DBA on board, general-purpose DBAs are still required to support the overall database environment and infrastructure. <h3>Task-oriented DBA</h3> This specialized DBA focuses on a specific administrative task and is uncommon outside of large IT shops. A backup and recovery DBA, for example, would be task-oriented to ensure the organization's databases are recoverable, including creating backup plans, building and testing backup scripts, testing recovery scripts, and driving recovery tasks when required. The backup and recovery DBA also participates in building and testing disaster <a href="https://www.techtarget.com/whatis/definition/contingency-plan">contingency plans</a> for the company's databases. <h3>Performance analyst</h3> As the most common task-oriented DBA, the performance analyst focuses entirely on monitoring databases and improving the performance of applications that access them. A performance analyst is an expert in <a href="https://www.techtarget.com/searchdatamanagement/definition/SQL">SQL coding</a> for performance and is knowledgeable in designing and building high-performance databases. Performance analysts should have a deep understanding of the DBMS, collaborate with other DBAs to execute changes when required and communicate with application developers in their language to facilitate appropriate program changes for performance. <h3>Data warehouse administrator</h3> This fully capable DBA has the knowledge and skills to monitor and support the data warehouse environment. Data warehouse administrators understand the differences between a database that supports <a href="https://www.techtarget.com/searchdatacenter/definition/OLTP">online transaction processing</a> and a data warehouse and must have experience in the following areas: <ul class="default-list"> <li><a href="https://www.techtarget.com/searchbusinessanalytics/definition/business-intelligence-BI">Business intelligence</a> and query tools.</li> <li>Specialized database design for data warehousing.</li> <li>ETL skills.</li> <li>Knowledge of data warehousing technologies, such as <a href="https://www.techtarget.com/searchdatamanagement/definition/OLAP">online analytical processing</a> and <a href="https://www.techtarget.com/searchdatamanagement/definition/star-schema">star schema</a>.</li> </ul> <h3>Cloud DBA</h3> As companies increasingly migrate workloads to the cloud, the cloud DBA has become more popular and performs many of the same tasks as a general-purpose DBA but for cloud database executions on services like <a href="https://www.techtarget.com/searchaws/definition/Amazon-Web-Services">Amazon Web Services</a> and <a href="https://www.techtarget.com/searchcloudcomputing/definition/Windows-Azure">Microsoft Azure</a>. The cloud DBA understands the services the <a href="https://www.techtarget.com/searchitchannel/definition/cloud-service-provider-cloud-provider">cloud provider</a> offers, including backup and security, to set up databases in the cloud. Cloud DBAs need to be aware of latency, fault tolerance and especially cost management because <a href="https://www.techtarget.com/searchdatamanagement/feature/Should-you-host-your-database-on-site-or-in-the-cloud">adding data or workloads to a cloud setup</a> can significantly increase costs. <h3>Data modelers</h3> Although an efficient database design must include <a href="https://www.techtarget.com/searchdatamanagement/definition/data-modeling">data modeling</a>, this isn't always considered the job of a DBA, since they're more focused on technology than the business. However, DBAs should know the essentials of data modeling, including how to build and manage data models to illustrate the relationships between data components. Data modelers create the data models and computer databases needed to transform intricate organizational data into functional computer systems. They also oversee the information flow across different departments within a business using relational, dimensional and <a href="https://www.techtarget.com/searchdatamanagement/definition/NoSQL-Not-Only-SQL">NoSQL</a> databases. </section> <section class="section main-article-chapter" data-menu-title="Data admin vs. database admin vs. system administrator"> <h2 class="section-title">Data admin vs. database admin vs. system administrator</h2> Although similar, the roles and responsibilities of a data administrator, <a href="https://www.techtarget.com/searchnetworking/definition/system-administrator">system administrator</a> and database administrator are different. The following is a brief overview of the tasks and responsibilities associated with each role and their points of distinction. <h3>Data administrator</h3> <ul class="default-list"> <li>Data administration separates the business aspects of data management from technology.</li> <li>Data administrators are aligned more with the business than with IT, translate the business lexicon into a logical data model and work with the DBA to translate models into actual databases.</li> <li>They generally own the data and manage tasks including data planning, definition, architecture and management.</li> <li>Data admins are responsible for ensuring the company's data practices are aligned with the relevant data privacy regulations.</li> </ul> <h3>System administrator</h3> <ul class="default-list"> <li>System administrators are responsible for the DBMS installation, configuration and setup but typically have no responsibility for database design and support.</li> <li>Their main job responsibilities entail managing the servers as well as the maintenance and operation of computer systems.</li> <li>They ensure the IT infrastructure is conducive to database development by setting up the DBMS appropriately, applying ongoing maintenance from the DBMS vendor and coordinating migration to new DBMS releases and versions.</li> </ul> <h3>Database administrator</h3> <ul class="default-list"> <li>Database administrators design and manage a database.</li> <li>They're responsible for the security, backup and recovery, and performance tuning of the database.</li> <li>DBAs own the database as well as enforce policies and procedures specific to the database.</li> <li>They regularly collaborate with business users to ensure the database meets their requirements and is performing efficiently.</li> <li>Duties of data admins and system admins typically fall into the laps of DBAs in organizations without data and system administrators.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="How to become a database administrator"> <h2 class="section-title">How to become a database administrator</h2> A database administrator is expected to stay abreast of emerging technologies and new design approaches. Typically, a DBA has a bachelor's degree in computer science or information systems from an accredited university or college as well as some on-the-job training with a specific database product. In some cases, DBAs might not be required to have a bachelor's degree if they have extensive IT work experience. Many DBAs have prior experience as application programmers and excelled at accessing databases using SQL embedded in <a href="https://www.techtarget.com/searchitoperations/definition/COBOL-Common-Business-Oriented-Language">COBOL</a>, or Common Business Oriented Language; <a href="https://www.theserverside.com/definition/Java">Java</a>; <a href="https://www.techtarget.com/searchwindowsserver/definition/C">C</a>; or other popular programming languages. DBAs are expected to be proficient in writing and debugging SQL. A DBA is usually expected to have experience with one or more of the following major database management products: <ul class="default-list"> <li>IBM <a href="https://www.techtarget.com/searchdatacenter/definition/DB2">Db2</a>.</li> <li><a href="https://www.techtarget.com/searchdatamanagement/definition/SQL-Server">Microsoft SQL Server</a>.</li> <li><a href="https://www.techtarget.com/searchdatamanagement/definition/MongoDB">MongoDB</a>.</li> <li>Oracle Database.</li> <li>PostgreSQL.</li> <li>Redis.</li> <li><a href="https://www.techtarget.com/searchoracle/definition/MySQL">MySQL</a>.</li> </ul> DBAs might also possess in-depth technical skills in related technologies, including DevOps software such as Docker, Kubernetes and Git; enterprise resource planning packages, such as <a href="https://www.techtarget.com/searchsap/definition/SAP">SAP</a>; operating systems, such as <a href="https://www.techtarget.com/searchdatacenter/definition/Linux-operating-system">Linux</a> and <a href="https://www.techtarget.com/searchdatacenter/definition/z-OS">z/OS</a>; and storage software. </section> <section class="section main-article-chapter" data-menu-title="Certifications for database administrators"> <h2 class="section-title">Certifications for database administrators</h2> Certification programs are available for most of the popular DBMS platforms and many organizations expect DBAs to be certified in the database systems they manage. Although not as important as on-the-job experience, DBAs who have kept current with their certifications should have relevant skills and knowledge about the features, functions and capabilities of the DBMS they're certified in. Some of the most in-demand database administrator certificates include the following: <ul class="default-list"> <li>Microsoft Certified: Azure Data Fundamentals. Entry-level database administrators who want to enhance their core knowledge of Azure can benefit greatly from this certification.</li> <li>Oracle Database Administration Certified Professional. This certification combines training, experience and testing to ensure a strong foundation and expertise in Oracle databases.</li> <li>MongoDB Associate Database Administrator. This certification is designed for those individuals who deal with NoSQL or non-relational databases at work, or who want to learn more about them.</li> <li>IBM Certified Administrator. This certification typically tests DBA skills and knowledge on the IBM Db2 product on the z/OS operating system.</li> <li>CompTIA DataSys+. This is a vendor-neutral, foundational-level data management certification designed for database administrators. It focuses on the logistics of data administration and security.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Salary range and employment outlook for DBAs"> <h2 class="section-title">Salary range and employment outlook for DBAs</h2> A database administrator's <a target="_blank" href="https://www.bls.gov/ooh/computer-and-information-technology/database-administrators.htm" rel="noopener">job</a> can be rewarding and well-compensated. According to the U.S. Bureau of Labor Statistics, the median annual salary for a DBA was $98,860 in 2023. Per Salary.com, the average database administrator salary in the U.S. is $101,796 as of March 26, 2024. But the salary range typically falls between $89,952 and $114,508, and it depends on factors such as experience, geographical location and industry. Another important consideration is employability. According to the U.S. Bureau of Labor Statistics "Occupational Outlook Handbook," employment of database administrators is projected to grow 8% from 2020 to 2030, which is about as fast as the average for all occupations. Database administration as a career is under pressure from <a href="https://www.techtarget.com/searchdatamanagement/tip/Cloud-database-comparison-AWS-Microsoft-Google-and-Oracle">cloud computing services</a>, such as the <a href="https://www.techtarget.com/searchdatamanagement/definition/database-as-a-service-DBaaS">database as a service</a> model, that provide some of the tasks traditionally provided primarily by DBAs, including database installation, provisioning, patching and some types of backup and performance monitoring. As a result, database administrators are increasingly more focused on applications than systems, because cloud service providers typically don't offer application-level services. In addition, DBAs are spending more time interfacing with developers, managing test data, problem-solving and optimizing performance than they do on installation and patching. <div class="youtube-iframe-container"> <iframe id="ytplayer-0" src="https://www.youtube.com/embed/f2M-dodlISQ?autoplay=0&modestbranding=1&rel=0&widget_referrer=null&enablejsapi=1&origin=https://searchsqlserver.techtarget.com" type="text/html" height="360" width="640" frameborder="0"></iframe> </div> Another nuance is the misconception that organizations don't need DBAs when they move data to the cloud. As DBA requirements shift more toward application support than system support, failing to staff DBAs for a <a href="https://www.techtarget.com/searchdatamanagement/tip/How-to-plan-and-manage-a-multi-cloud-database-environment">cloud database infrastructure</a> can result in inefficient applications, insecure data and perhaps exorbitant cloud service provider costs. The role of a database administrator is evolving rapidly due to the widespread use of cloud computing. Find out what makes the role of a <a href="https://www.techtarget.com/searchdatamanagement/tip/Cloud-DBA-How-cloud-changes-database-administrators-role">cloud DBA different from an on-premises DBA</a>. </section> A database administrator (DBA) is the information technician responsible for directing and performing all activities related to maintaining and securing a successful database environment. https://cdn.ttgtmedia.com/visuals/digdeeper/3.jpg https://www.techtarget.com/searchdatamanagement/definition/database-administrator Wed, 01 May 2024 14:43:00 GMT database administrator (DBA) <h3>What is database replication?</h3> Database replication is the frequent electronic copying of <a href="https://www.techtarget.com/searchdatamanagement/definition/data">data</a> from a database in one computer or <a href="https://www.techtarget.com/whatis/definition/server">server</a> to a database in another -- so that all users share the same level of information. The result is a <a href="https://www.techtarget.com/searchoracle/definition/distributed-database">distributed database</a> in which users can quickly access data relevant to their tasks without interfering with the work of others. Numerous elements contribute to the overall process of creating and managing database replication. <h3>How database replication works</h3> Database replication can either be a single occurrence or an ongoing process. It involves all data sources in an organization's distributed infrastructure. The organization's distributed management system is used to replicate and properly distribute the data amongst all the sources. Overall, distributed database management systems (<a href="https://www.techtarget.com/searchoracle/definition/distributed-database">DDBMS</a>) work to ensure that changes, additions and deletions performed on the data at any given location are automatically reflected in the data stored at all the other locations. DDBMS is essentially the name of the infrastructure that allows or carries out database replication -- the system that manages the distributed database, which is the product of database replication. The classic case of database replication involves one or more applications that connect a primary storage location with a secondary location that is often off site. Today, those primary and secondary storage locations are most often individual source databases -- such as Oracle, MySQL, Microsoft SQL and MongoDB -- as well as data warehouses that amalgamate data from these sources, offering storage and analytics services on larger quantities of data. Data warehouses are often hosted in the cloud. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/data_management-database_replication.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/data_management-database_replication_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/data_management-database_replication_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/data_management-database_replication.png 1280w" alt="database replication diagram" height="304" width="520"> <figcaption> Database replication copies data from a principal location. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> <h3>Database replication techniques</h3> There are several ways to replicate a database. Different techniques offer different advantages, as they vary in thoroughness, simplicity and speed. The ideal choice of technique depends on how companies store data and what purpose the replicated information will serve. Regarding the timing of data transfer, there are two types of data replication: <ul class="default-list"> <li>Asynchronous replication is when the data is sent to the model server -- the server that the replicas take data from -- from the client. Then, the model server pings the client with confirmation saying the data has been received. From there, it goes about copying data to the replicas at an unspecified or monitored pace.</li> <li>Synchronous replication is when data is copied from the client server to the model server and then replicated to all the replica servers before the client is notified that data has been replicated. This takes longer to verify than the asynchronous method, but it presents the advantage of knowing that all data was copied before proceeding.</li> </ul> <a href="https://www.techtarget.com/searchdatabackup/definition/asynchronous-replication">Asynchronous database replication</a> offers flexibility and ease of use, as replications happen in the background. However, there is a greater risk that data will be lost without the client's knowledge because confirmation comes before the main replication process. <a href="https://www.techtarget.com/searchdisasterrecovery/definition/synchronous-replication">Synchronous replication</a> is more rigid and time-consuming, but more likely to ensure that data will be successfully replicated. The client will be alerted if it hasn't, since confirmation comes after the entire process has finished. There are also several types of database replication based on the type of server architecture. The term leader will be used in these types to mean the same thing as model in the previous asynchronous vs. synchronous examples: <ul class="default-list"> <li>Single-leader architecture is one server that receives writes from clients, and replicas draw data from there. This is the most common and classic method. It's a synchronized method, but somewhat inflexible.</li> <li>Multi-leader architecture is multiple servers that can receive writes and serve as a model for replicas. It is beneficial for when replicas are spread out and leaders must be near all of them to prevent latency.</li> <li>No-leader architecture is every server that can receive writes and serve as a model for replicas. This was pioneered by Amazon's DynamoDB. While it offers maximum flexibility, it poses challenges for synchronization.</li> </ul> <h3>Advantages and disadvantages</h3> Database replication is often overseen by a database or replication administrator. A properly implemented replication system can offer several advantages, including the following: <ul class="default-list"> <li>Load reduction. Because replicated data can be spread over several servers, it eliminates the likelihood that any one server will be overwhelmed with user queries for data.</li> <li> Efficiency. Servers that are less burdened with queries can offer improved performance to fewer users.</li> <li>High availability. Employing multiple servers with the same data ensures high availability, meaning that if one server goes down, the entire system can still provide acceptable performance.</li> </ul> Many disadvantages of database replication stem from poor <a href="https://www.techtarget.com/searchdatamanagement/feature/Evaluate-your-data-governance-and-privacy-best-practices">general data governance practices</a>. These disadvantages include the following: <ul class="default-list"> <li>Data loss. Data loss can occur during replication when incorrect data or iterations or updates of a database are copied and, consequently, important data is deleted or unaccounted for. This can happen if the primary key used to verify the quality of data in the replica is malfunctioning or incorrect. It can also occur if database objects are incorrectly configured within the source database.</li> <li>Data inconsistency. Similarly, incorrect or out-of-date replicas can cause different sources to be out of sync with each other. This may lead to wasted data warehouse costs that are spent needlessly analyzing and storing irrelevant data.</li> <li>Multiple servers. Running multiple servers has an inherent maintenance and energy cost associated. It requires either the organization or a third party to address these costs. If a third party handles them, the organization runs the risk of vendor lock-in or service issues beyond the organization's control.</li> </ul> <h3>Evolution of database replication</h3> Early instances of database replication were typically described as master-slave configurations, but comparable descriptions today tend to incorporate terminology such as master-replica, leader-follower, primary-secondary and server-client. Replication techniques centered on <a href="https://www.techtarget.com/searchdatamanagement/definition/relational-database">relational database</a> management systems have expanded with the advent of the <a href="https://searchservervirtualization.techtarget.com/definition/virtual-machine">virtual machine</a> and distributed cloud computing, to include nonrelational database types. Again, replication methods vary among such nonrelational databases as Redis, <a href="https://www.techtarget.com/searchdatamanagement/definition/MongoDB">MongoDB</a> and the like. While remote office database replication may have been the canonical example of replication for many years, fail-safe and <a href="https://www.techtarget.com/searchdisasterrecovery/definition/fault-tolerant">fault-tolerant</a> database backup schemes have also arisen as drivers of replication activity -- as have horizontally scaling distributed database configurations, both on premises and on <a href="https://www.techtarget.com/searchcloudcomputing/definition/cloud-computing">cloud computing</a> platforms. Replication details vary between such relational systems as IBM Db2, Microsoft SQL Server, Sybase, <a href="https://www.techtarget.com/searchoracle/definition/MySQL">MySQL</a> and PostgreSQL. In all cases, data replication design becomes a balancing act between system performance and data consistency. Database replication can be done in at least three different ways. In snapshot replication, data on one server is simply copied to another server or to another database on the same server. In merging replication, data from two or more databases is combined into a single database. And, in <a href="https://www.techtarget.com/searchcio/definition/transaction">transactional</a> replication, user systems receive full initial copies of the database and then receive periodic updates as data changes. <div class="youtube-iframe-container"> <iframe id="ytplayer-0" src="https://www.youtube.com/embed/sT2RQKlhU20?autoplay=0&modestbranding=1&rel=0&widget_referrer=null&enablejsapi=1&origin=https://searchsqlserver.techtarget.com" type="text/html" height="360" width="640" frameborder="0"></iframe> </div> <h3>Database replication vs. mirroring</h3> While data mirroring is sometimes positioned as an alternative approach to data replication, it is actually a form of data replication. In relational database mirroring, complete backups of databases are maintained for use in the case that the primary database fails. Mirrors, in effect, serve as hot standby databases. Data mirroring has found considerable use within the Microsoft <a href="https://www.techtarget.com/searchdatamanagement/definition/SQL-Server">SQL Server</a> community. With database replication, the focus is usually on database scale out for <a href="https://searchsqlserver.techtarget.com/definition/query">queries</a> -- requests for data. Database mirroring, in which log extracts form the basis for incremental database updates from the principal server, is typically implemented to provide hot standby or <a href="https://www.techtarget.com/searchdisasterrecovery/definition/disaster-recovery">disaster recovery</a> capabilities. Simply put, mirroring focuses on backing up what's there, and replication focuses on improving operational efficiency as a whole -- which involves maintaining secure data backups using mirroring. <h3>Database replication tools</h3> Companies can either use the database replication tool available offered by their database software provider or invest in third-party replication tools to execute and manage database replication processes. The latter option allows flexibility: Third-party tools are typically vendor-agnostic and can be used to create data replicas across multiple types of databases in an organization. <h3>Database replication software</h3> Third-party database replication tools that work with various databases include the following: <ul class="default-list"> <li>Qlik Replicate. A software package that focuses on being easy to learn and implement, Replicate uses automation and log-based capture to minimize IT operations workload. This involves capturing streams of continuous data, which is ideal for companies that need ways to process big data efficiently.</li> <li>Informatica Data Replication. Informatica can target a wide range of database and data warehouse appliances. It offers the Data Engineering product series for streaming, integration, quality and masking enterprise data. Its website features a how-to library and list of guides to help customers.</li> <li>Talend Open Studio for Data Integration. One of the most prominent open source data integration products, it includes a large repertoire of resources for those just getting started with the software. Talend offers tutorials, demos and blog posts on topics like the use of metadata and best practices for data model design best practices. There is also a community of experienced users that new users can reach out to for tips on using Talend's data integration solution.</li> <li>Quest SharePlex. This offering focuses mainly on Oracle database replication. It offers both on-premises and cloud solutions for Oracle database replication. Shareplex promises high availability, 24/7 customer support and a simple user experience that allows for quick replication and scaling.</li> </ul> Examples of database vendor replication tools include the following: <ul class="default-list"> <li>Microsoft's SQL integration features. Including SQL Server Integration Services (SSIS) for Azure, these tools specialize in cleaning, aggregating, merging and copying data, as well as extracting and loading data.</li> <li>Oracle GoldenGate. This tool offers log-based capture for Oracle databases. It promises simple configuration, extreme performance and comprehensive security. It features a visual management and monitoring feature called the Management Pack as well.</li> <li>IBM'S Db2 SQL replication tool. This tool has two main replication options: Q and SQL. It is one of IBM's most popular replication tools. It is useful for distributing source data to multiple targets, but may not be ideal for all replication scenarios because it has high latency.</li> </ul> Database replication is the frequent electronic copying of data from a database in one computer or server to a database in another -- so that all users share the same level of information. https://cdn.ttgtmedia.com/visuals/digdeeper/6.jpg https://www.techtarget.com/searchdatamanagement/definition/database-replication Thu, 29 Dec 2022 16:45:00 GMT database replication Microsoft SQL Server is a relational database management system that can support a wide range of applications, but the data must be readily available to those applications whenever they need it. Any disruptions to databases services -- whether from natural disaster, equipment failure, cyber attacks or other causes -- can prevent an organization from performing routine operations and conducting everyday business. This may lead to disgruntled users, lost revenue and tarnished reputations. An organization must have an effective disaster recovery strategy that helps to minimize disruptions to <a href="https://www.techtarget.com/searchdatamanagement/definition/SQL-Server">SQL Server</a> services, especially when supporting mission-critical workloads. Organizations of all types and sizes use SQL Server to support transaction processing, business intelligence and analytics applications. <h3>Planning for the worst</h3> Like similar database products, such as Oracle Database or IBM's Db2, SQL Server is built on the Structured Query Language (SQL), a standardized programming language that provides a foundation for managing relational databases and querying their data. SQL Server uses a modified implementation of SQL -- referred to as Transact-SQL -- that adds a set of proprietary programming extensions to the standard language. Applications that rely on SQL Server must be able to access the data to meet workload requirements, but unexpected events can cause databases to become unavailable or lead to data loss. Disruptions to data services can occur for numerous reasons, including the following: <ul class="default-list"> <li>a user inadvertently deletes critical data;</li> <li>a <a href="https://www.techtarget.com/searchsecurity/post/5-questions-to-ask-when-creating-a-ransomware-recovery-plan">malware attack</a> encrypts or destroys data;</li> <li>an employee spills coffee on the storage system;</li> <li>a power failure causes data to become corrupted;</li> <li>an earthquake destroys physical storage infrastructure;</li> <li>a hard disk drive (HDD) fails and all data on that drive is lost;</li> <li>an administrator inadvertently formats an HDD that contains active data;</li> <li>software corruption leads to data loss or lack of availability; and</li> <li>a rogue employee steals a physical storage device.</li> </ul> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> SQL Server has features that can help protect against disaster, many of which are part of the platform's high-availability capabilities. </figure> </div> </blockquote> These are by no means the only reasons that data might become unavailable but illustrate the wide range of crises that could strike any organization at any time. Whatever the cause, the only way to protect against data loss and service disruptions is to <a href="https://www.techtarget.com/searchdisasterrecovery/feature/IT-disaster-recovery-DR-plan-template-A-free-download-and-guide">implement a disaster recovery strategy</a> that minimizes downtime and ensures the safe recovery of any affected data. Fortunately, SQL Server has features that can help protect against disaster, many of which are part of the platform's high-availability capabilities. The primary goal of a DR plan is to ensure business continuity if disaster strikes. Not all plans are the same, however, nor should they be. A DR plan should be tailored to meet an organization's specific data protection requirements. For example, data that drives a mission-critical financial application might require immediate recovery, but data that's used to generate monthly sales reports can likely tolerate a longer delay. When planning a DR strategy, an organization should determine the following three metrics: <ul class="default-list"> <li>Recovery Time Objective (RTO). This is the maximum amount of time an application can be offline as a result of unavailable data. The metric determines how quickly data needs to be back online after an incident.</li> <li>Recovery Point Objective (RPO). This is the acceptable amount of data loss that can be tolerated if an event occurs. This metric is often considered in terms of time. For example, if a database is backed up every two hours and a disaster occurs right before the next scheduled backup, all data changes that took place since the last backup are lost.</li> <li>Recovery Level Objective (RLO). This is the level of granularity at which the data should be recovered, such as the instance, database or table level.</li> </ul> An organization will ultimately have to balance the risk of data loss against the costs of implementing a DR strategy. The shorter the <a href="https://www.techtarget.com/searchstorage/feature/What-is-the-difference-between-RPO-and-RTO-from-a-backup-perspective">RTO and RPO</a> (in terms of time) and the finer the <a href="https://www.techtarget.com/searchdisasterrecovery/tip/Use-recovery-level-objective-to-fine-tune-BCDR-response">RLO granularity</a>, the greater the costs. <h3>Disaster recovery strategies for SQL Server</h3> After organizations determine the levels of protection they need, they can take advantage of the following SQL Server features to help mitigate the impact of unexpected service disruptions: <ul class="default-list"> <li>Backup and recovery. A well-tested backup and restore strategy can help protect databases against a variety of issues, including ransomware, user error, hardware failures and natural disaster. Backing up data to a separate location makes it possible to <a href="https://www.techtarget.com/searchdatabackup/answer/What-are-some-best-practices-for-planning-database-backups">restore a database</a> to a consistent state and avoid catastrophic data loss; however, restoring databases can be a time-consuming process and data changes made after the last backup are lost. Backups have become especially important with the rise in ransomware attacks.</li> <li>Always On availability groups. Like backups, availability groups provide database-level protection. An availability group is made up of a group of databases that fail over together to a secondary instance. SQL Server can support up to <a href="https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/overview-of-always-on-availability-groups-sql-server?view=sql-server-ver15" target="_blank" rel="noopener">eight sets</a> of corresponding secondary databases in an availability group. Within the availability group, a database's transactions are applied to a replica database on another SQL Server instance. Availability groups serve as a replacement to database mirroring, which is still supported but was deprecated in SQL Server 2012.</li> <li>Always On failover cluster instances. Failover clusters use the <a href="https://www.techtarget.com/searchwindowsserver/tutorial/Build-a-Windows-Server-2019-file-server-cluster-for-stability">Windows Server Failover Cluster (WSFC)</a> nodes framework to provide instance-level protection that includes databases, linked servers, SQL Server Agent jobs and other instance and database objects. A failover cluster is made up of redundant nodes, but only one node can own the WSFC resource group at a time. A cluster also requires a shared storage system, such as a storage area network. Failover is automatic and transparent to connecting applications.</li> <li>SQL Server replication. Replication makes it possible to copy and distribute data and database objects from one database to another and then keep those databases synchronized. SQL Server supports three types of replication: transactional, snapshot and merge. Organizations can use replication to distribute data to different locations across local area networks, wide area networks, wireless connections and the internet. SQL Server replication uses SQL Server Agent to synchronize the databases and their data.</li> <li>Log shipping. Log shipping enables administrators to automatically apply a database's transaction log on one SQL Server instance to one or more secondary databases on separate instances. A monitor server maintains a record of the history and status of backup and restore operations and raises alerts if operations fail. Log shipping uses SQL Server's backup and restore capabilities to copy the transaction log from the primary instance to the secondary instances. Although log shipping is easy to implement, switching over to a secondary database is a manual operation that can take a significant amount of time.</li> </ul> Many organizations deploy these features in combination to help maximize data protection and minimize downtimes. For example, a database team might use log shipping along with availability groups to protect their databases. In addition, most organizations maintain backups regardless of any other strategies they deploy. Many organizations also implement SQL Server disaster recovery as part of a larger DR strategy that incorporates protections at multiple levels, such as configuring <a href="https://www.techtarget.com/searchdatabackup/tip/RAID-5-vs-RAID-6-Capacity-performance-durability">RAID 6 storage</a> or backing up virtual machines running SQL Server instances. Regardless of the DR strategy that an organization implements, it first requires careful planning that considers RTO, RPO and RLO requirements, as well as factors such as security and compliance. Disaster recovery also goes hand in hand with an organization's high-availability strategy, which relies on many of the same SQL Server tools. Regardless of an organization's type or size, disaster recovery should be a top priority, and the more prepared organizations are to handle disasters, the more likely they are to get through them when they occur. SQL Server offers a platform for hosting relational databases. Enacting a disaster recovery plan that won't disrupt mission-critical SQL Server workloads is essential. https://cdn.ttgtmedia.com/rms/onlineimages/maze_g1261411056.jpg https://www.techtarget.com/searchdisasterrecovery/feature/How-to-manage-disaster-recovery-in-SQL-Server Fri, 25 Feb 2022 15:46:00 GMT How to manage disaster recovery in SQL Server The landscape for database management systems (<a href="https://searchsqlserver.techtarget.com/definition/database-management-system">DBMSes</a>) continues to evolve to adapt to changing requirements for transaction processing, data access and big data applications. One distinguishing characteristic of modern DBMS offerings is in their <a href="https://www.techtarget.com/searchdatamanagement/definition/SQL">SQL</a> capabilities. At a high level, it is possible to categorize most modern DBMS products into one of the following designations: <ul class="default-list"> <li>SQL</li> <li><a href="https://www.techtarget.com/searchdatamanagement/definition/NoSQL-Not-Only-SQL">NoSQL</a></li> <li>NewSQL</li> </ul> But these terms can contain multiple layers of meaning. To further complicate matters, they are not mutually exclusive. <section class="section main-article-chapter" data-menu-title="What are SQL and relational database systems?"> <h2 class="section-title">What are SQL and relational database systems?</h2> At a high-level SQL, or Structured Query Language, is the standard query language for interacting with a relational database management system (<a href="https://www.techtarget.com/searchdatamanagement/definition/RDBMS-relational-database-management-system">RDBMS</a>). Indeed, every RDBMS -- and many nonrelational DBMS products -- supports SQL as the method for accessing data. Although relational and SQL are used somewhat analogously, there are differences. While the differences are more nuanced, a traditional SQL database system can be thought of as being based on the relational model. The relational model relies on set theory and relations to describe data in the database. From an external viewpoint, this means data is viewed as tables of rows and columns using a fixed <a href="https://searchsqlserver.techtarget.com/definition/schema">schema</a>, meaning each row has the same columns defined using the same data types as every other row in a table. Each table row has a key that uniquely identifies the row. For example, the Product table key may be Product ID. Relationships between tables are defined by the values of columns. For example, the Order table is related to the Product table by means of the Product ID column, which must exist in both tables. Furthermore, SQL database systems are typified by their support of <a href="https://searchsqlserver.techtarget.com/definition/ACID">ACID</a> (atomicity, consistency, isolation and durability) support for transactions. A DBMS with ACID support generally means data consistency is strictly maintained as multiple processes access and modify the same data. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineImages/data_management-dbms_vs_rdbms.jpg"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineImages/data_management-dbms_vs_rdbms_mobile.jpg" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineImages/data_management-dbms_vs_rdbms_mobile.jpg 960w,https://searchsqlserver.techtarget.com/rms/onlineImages/data_management-dbms_vs_rdbms.jpg 1280w" alt="Differences between DBMS and RDBMS" height="280" width="560"> <figcaption> Review the differences between DBMSes and RDBMSes. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> <h3>Advantages and disadvantages of SQL database systems</h3> Advantages of using SQL database systems include: <ul class="default-list"> <li>SQL database systems are pervasive and fit well for general-purpose data storage, management and access requirements.</li> <li>All data access is accomplished using a single, standardized language.</li> <li>SQL is portable across different DBMS products with minimal changes.</li> <li>SQL database systems have a 40-year history of success, and, as such, there are many users of SQL database systems, making it is easier to find skilled professionals to develop and administer these databases.</li> </ul> Disadvantages of using SQL database systems include: <ul class="default-list"> <li>Relational databases <a href="https://www.theserverside.com/blog/Coffee-Talk-Java-News-Stories-and-Opinions/How-to-create-a-database-schema-with-the-MySQL-Workbench">need a fixed schema</a>, as they sometimes don't work with modern development. RDBMSes are not well suited to work with niche areas in comparison to NoSQL database systems.</li> <li>Because of the ACID consistency model, SQL databases can't scale horizontally, but they can scale vertically by increasing CPU and memory.</li> </ul> <h3>Examples of SQL database systems</h3> Many of the most popular database systems in use today are based on relational and SQL, including: <ul class="default-list"> <li>Oracle Database</li> <li>Microsoft SQL Server</li> <li>MySQL</li> <li>IBM Db2</li> <li>PostgreSQL</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="What is NoSQL?"> <h2 class="section-title">What is NoSQL?</h2> As data volume has grown over the past few decades, SQL databases have struggled with particular use cases driven by big data requirements. NoSQL (Not Only SQL) database systems grew in popularity to support use cases, such as: <ul class="default-list"> <li>Profile management</li> <li>Content management</li> <li>Mobile application</li> <li>Real-time usage of structured and unstructured data</li> <li>Social media management</li> </ul> NoSQL suggests nonrelational, distributed, flexible and scalable. Many NoSQL database systems are also open source. Additionally, some common features of NoSQL DBMSes include data clustering, lack of a fixed schema, replication support and eventual consistency, which is in opposition to the usual ACID transaction capability of SQL database systems. An eventually consistent approach tolerates inconsistent data across nodes for a period of time. However, as the name implies, data integrity is eventually enforced across all the nodes. Another big difference between NoSQL and relational/SQL is that NoSQL systems do not require SQL for accessing data. However, many NoSQL databases have added support for SQL. Whereas SQL database systems require a rigidly defined schema, most NoSQL databases implement flexible schemas, wherein each instance of data (e.g., table row) need not contain the same data elements and definition. When data structures can evolve over time, a NoSQL database system can be more practical than a SQL/relational database system. There are no hard-and-fast rules as to how NoSQL databases store data. There are actually <a href="https://www.techtarget.com/searchdatamanagement/infographic/NoSQL-database-comparison-to-help-you-choose-the-right-store">multiple types of DBMS platforms categorized as NoSQL</a>, which can be viewed as an overarching descriptor for four different types of database systems: <ul class="default-list"> <li>Key-value databases store pairs of unique keys and associated values.</li> <li>Document databases <a href="https://www.techtarget.com/searchdatamanagement/tip/NoSQL-database-types-explained-Document-based-databases">store data in document-like structures</a> encoded in formats such as JSON and XML.</li> <li>Wide-column databases store data in tables that contain large numbers of columns (in a flexible schema).</li> <li>Graph databases <a href="https://www.techtarget.com/searchdatamanagement/tip/NoSQL-database-types-explained-Graph">store data in graph form</a> to highlight the connections between different data elements.</li> </ul> Each type of NoSQL DBMS is best suited to particular use cases and has individual pros and cons to consider. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineImages/data_management-nosql.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineImages/data_management-nosql_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineImages/data_management-nosql_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineImages/data_management-nosql.png 1280w" alt="Types of NoSQL database systems" height="336" width="560"> <figcaption> There are four different types of NoSQL database systems, each used handle data differently. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> <h3>Advantages and disadvantages of NoSQL database systems</h3> NoSQL database systems have the following advantages: <ul class="default-list"> <li>NoSQL database systems simplify some types of application development, such as interactive real-time web applications using a <a href="https://www.techtarget.com/searchapparchitecture/definition/RESTful-API">representational state transfer application programming interface</a>and web services.</li> <li>They support flexibility for data that has not been normalized, which requires a flexible data model or has different properties for different data entities.</li> <li>NoSQL database systems deliver scalability for larger data sets, which are common in analytics and artificial intelligence applications.</li> <li>NoSQL is <a href="https://www.techtarget.com/searchcloudcomputing/tip/Compare-NoSQL-database-types-in-the-cloud">better suited for cloud</a>, mobile, social media and big data requirements.</li> <li>They are easier to use than general-purpose SQL databases for the use cases for which they are designed.</li> </ul> Disadvantages of NoSQL database systems include: <ul class="default-list"> <li>Each NoSQL database has its own syntax for querying and managing data. This contrasts with SQL, which is standard for all SQL database systems.</li> <li>Lack of a rigid database schema and constraints removes the data integrity safeguards that are built into SQL database systems.</li> <li>A schema with some sort of structure is required in order to use the data. With NoSQL, this must be performed by the application developer instead of the <a href="https://searchsqlserver.techtarget.com/definition/database-administrator">database administrator</a>.</li> <li>Eventual consistency is typically acceptable only for fault-tolerant applications, as it does not provide the same level of data consistency as the ACID support built into SQL databases.</li> <li>Because NoSQL is newer, there are no comprehensive industry standards as with relational and SQL DBMS offerings. Additionally, it can be more difficult to find skilled NoSQL professionals.</li> </ul> <h3>Examples of NoSQL database systems</h3> There is a plethora of popular NoSQL database systems used for production workloads today, including: <ul class="default-list"> <li>MongoDB, a document database</li> <li>Redis, a key/value database</li> <li>Cassandra, a wide-column database</li> <li>Neo4j, a graph database</li> </ul> <div class="youtube-iframe-container"> <iframe id="ytplayer-0" src="https://www.youtube.com/embed/KxLmm7a-4aI?autoplay=0&modestbranding=1&rel=0&widget_referrer=null&enablejsapi=1&origin=https://searchsqlserver.techtarget.com" type="text/html" height="360" width="640" frameborder="0"></iframe> </div> </section> <section class="section main-article-chapter" data-menu-title="What is NewSQL?"> <h2 class="section-title">What is NewSQL?</h2> Yet another type of database system, NewSQL, is a class of modern relational/SQL DBMS that provides the same scalable performance of NoSQL systems for online workloads but also provides ACID so as to not sacrifice data consistency. The term was first used by 451 Research in a 2011 research paper discussing the rise of new database systems as challengers to established vendors. Basically, a NewSQL DBMS is engineered as a relational, SQL database system with a distributed, fault-tolerant architecture. Other typical features of NewSQL offerings include in-memory capability and clustered database services with the ability to be deployed in the cloud. Many NewSQL DBMS packages have fewer features and components and a smaller footprint than legacy relational offerings, making them easier to support and understand. The general use case for NewSQL is to support enterprise systems that must process large amounts of data (big data) while maintaining transactional consistency and are, therefore, not practical for the eventual consistency of NoSQL database systems. There are several different approaches taken by NewSQL database systems to achieve higher scalability with ACID consistency. Many NewSQL database systems are built on new, modern architectures that were not conceivable when the earliest SQL database systems were developed in the late 1970s. For example, many NewSQL offerings deploy a cluster of shared-nothing nodes, each managing a subset of the data with distributed concurrency control and query processing to balance the workload. Another approach taken by NewSQL vendors is to transparently shard data across multiple nodes using a <a href="https://www.techtarget.com/whatis/definition/consensus-algorithm">consensus algorithm</a>. And most NewSQL database systems deploy improved SQL engines for data storage and optimization. <h3>Advantages and disadvantages of NewSQL</h3> Advantages of NewSQL database systems include: <ul class="default-list"> <li>NewSQL works well for applications having many short transactions that access a small amount of indexed data and execute repetitively.</li> <li>NewSQL uses SQL, the most common data access language and, therefore, provides familiarity for developers and programmers.</li> <li>NewSQL supports ACID transaction consistency, thereby providing better data integrity than most NoSQL alternatives.</li> <li>NewSQL can reduce complexity for applications that do not require all the bells and whistles of a traditional SQL DBMS.</li> </ul> Disadvantages of NewSQL database systems include: <ul class="default-list"> <li>NewSQL has fewer features than traditional SQL and, therefore, NewSQL is not as capable for general purpose implementation.</li> <li>The lack of tools to support development and administration as compared to traditional SQL database systems.</li> <li>There are fewer skilled professionals available due to NewSQL's newer technology.</li> </ul> <h3>Examples of NewSQL database systems</h3> An increasing number of NewSQL DBMS offerings are being implemented for production workloads, including: <ul class="default-list"> <li>CockroachDB</li> <li>VoltDB</li> <li>NuoDB</li> <li>SingleStore</li> <li>Clustrix</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="How to choose among SQL vs. NoSQL vs. NewSQL"> <h2 class="section-title">How to choose among SQL vs. NoSQL vs. NewSQL</h2> After reviewing the basics of each type of database system, the question of <a href="https://www.techtarget.com/searchdatamanagement/feature/Evaluating-the-different-types-of-DBMS-products">which DBMS to choose</a> may remain. It makes sense to use SQL database systems as the baseline for most general-purpose requirements. There is a reason the SQL DBMS is as popular and entrenched as it is: It works very well for many of the most common use cases. For applications with extreme availability and scalability needs or with unknown or changing data requirements where a flexible schema would be useful, consider a NoSQL database system. But take heed of the potential for data consistency issues and realize that NoSQL is a term that describes four different types of database systems, each with different use cases and architectures. Finally, consider a NewSQL database system for applications that have expanded past the scalability capability of traditional SQL. Using NewSQL enables improved scalability with the strong transaction consistency of ACID, while still supporting data access using the familiar SQL. </section> Choosing which database management system depends on an organization's needs. Review the differences among SQL, NoSQL and NewSQL and how to select the right database system. https://cdn.ttgtmedia.com/visuals/searchDataManagement/governance/datamanagement_article_012.jpg https://www.techtarget.com/whatis/feature/SQL-vs-NoSQL-vs-NewSQL-How-do-they-compare Fri, 30 Jul 2021 14:16:00 GMT SQL vs. NoSQL vs. NewSQL: How do they compare? Windows Server has evolved over the years, and tweaks to keep the operating system running well that were valid years ago might not hold up. What follows are 10 basic pointers that can help boost Windows Server performance. While one change alone might not make much of a difference, collectively, these adjustments can give your workload the pep it needs. <h3>Add SSD storage</h3> Even if it isn't practical to run a server entirely on flash storage, it's a good idea to use solid-state drive (SSD) storage for the server's boot volume and for any application binaries. This will go a long way toward <a href="https://www.techtarget.com/searchwindowsserver/tip/Windows-Server-2022-storage-features-address-security-speed">improving the server's overall responsiveness</a>. While SAS or SATA SSD drives are improvements over spinning hard drives, an enterprise-grade SSD drive that uses the <a href="https://www.techtarget.com/searchstorage/feature/NVMe-SSD-speeds-explained">NVMe protocol will bring even more benefits</a> that will maximize Windows Server performance, including lower system overhead to reduce latency. <h3>Use the Resilient File System</h3> Microsoft developed the Resilient File System (ReFS) as the potential successor to the New Technology File System with improvements in data integrity and support for larger volumes and file sizes. Under the right conditions, <a href="https://www.techtarget.com/searchwindowsserver/answer/Whats-the-difference-between-FAT32-FAT16-and-NTFS">ReFS can also give your server a performance boost</a>. For example, the file system's block cloning feature can speed up checkpoint-related functions for Hyper-V virtual machines. Likewise, the sparse valid data length feature accelerates the creation of fixed-length virtual hard disks from several minutes to several seconds. <h3>Update the server's firmware</h3> Physical servers should always have the latest firmware, as updates can sometimes improve the server's performance and stability. There is an issue that affects some Windows Server deployments that use a balanced power plan and may suffer from poor performance. Oftentimes, this issue stems from a server on outdated firmware. <h3>Look for memory leaks</h3> Poorly written applications can cause what are known as memory leaks. An application looks for memory from the operating system and, when it is no longer needed, releases that memory back to the system. A memory leak occurs when the application does not relinquish the memory, even when it is no longer needed, and will request more memory despite having enough in reserve. This will gradually reduce the amount of available memory to other applications and, eventually, cause overall performance to suffer. The Performance Monitor -- sometimes referred to by its executable name, perfmon -- is one tool that helps uncover these memory leaks. Microsoft recently updated the tool after 25 years and included it in the <a href="https://www.techtarget.com/searchwindowsserver/feature/Windows-Admin-Center-brings-server-management-under-one-roof">Windows Admin Center</a>. You can collect data for a memory test by setting up a performance counter in the Performance Monitor to track memory usage to determine where the issue lies. If the application is found to be the source of the memory leak, then it's time to bring your findings to the vendor for a fix. <h3>Jettison the page file</h3> Microsoft designed the Windows page file -- also called the paging file -- to make up for a shortage in physical memory on the server. While Windows can still use a page file for this purpose, the page file is more commonly used for a system crash dump file. In many instances, the page file is no longer needed. From a performance standpoint, it is better to make sure that a server has the memory it needs, rather than relying on a page file to make up for inadequate RAM. If you do decide to use a page file, it should reside on a fast disk. Check the <a href="https://docs.microsoft.com/en-us/windows/client-management/determine-appropriate-page-file-size" target="_blank" rel="noopener">following documentation</a> from Microsoft to size the page file appropriately. <h3>Run only what you need</h3> As a general rule, you should only run one application per server or VM. This practice makes it easier to troubleshoot problems but also helps system performance because multiple applications will compete for hardware resources. It's relatively common for a Windows Server workload to require some supplementary applications, such as antivirus software, backup agents or management agents. You should only run essential tools to maximize the amount of processing power and memory for business workloads. <h3>Disable unused services</h3> Use the service control manager built into Windows Server and disable any services that run in the background and are not required for the server role. This practice helps to boost both Windows Server performance and system security. The more code that's actively running on a system, the larger the <a href="https://www.techtarget.com/whatis/definition/attack-surface">attack surface</a> available to an attacker. Most Windows services on the server run with elevated privileges, so any vulnerability in a service has the potential to be exploited. <h3>Get rid of the GUI</h3> While some applications such as Exchange Server 2016 require a Windows Server GUI to function, many do not. Microsoft calls the GUI version Windows Server with Desktop Experience on Windows Server 2016 and up. On Windows Server 2012 and 2012 R2 systems, Microsoft calls this feature Windows Server with a GUI. Windows Server without the GUI is called Server Core. If you find that a server's workload can get by without a GUI, then you can streamline the operating system's resource consumption by uninstalling the GUI. This doesn't mean that you have to resort to managing the server through PowerShell; although, that is an option. You can simply run the <a href="https://www.techtarget.com/searchwindowsserver/tip/CLI-vs-GUI-Find-the-right-admin-tool-for-various-scenarios">GUI-based management tools</a> from a remote machine. To remove the GUI, open an elevated PowerShell session and run the commands below, which only works on Windows Server 2012 and 2012 R2 systems. On these newer Windows Server versions, you need to start with a Server Core deployment because Microsoft removed the ability to add and remove the GUI starting with Windows Server 2016. <pre>Uninstall-WindowsFeature Server-Gui-Shell Uninstall-WindowsFeature Server-Gui-Mgmt-Infra –Restart</pre> If you later decide that you need to reinstall the GUI, you can do so by entering this command: <pre>Install-WindowsFeature Server-Gui-Shell, Server-Gui-Mgmt-Infra –Restart</pre> <h3>Update the hardware device drivers</h3> One of the most important things you can do to keep Windows Server performance at an optimal level is to keep the device drivers up to date, assuming that the OS is running on physical hardware. Server vendors typically release updated device drivers on a regular basis, which often improve performance. Microsoft sometimes also supplies drivers, but it's usually best to use the ones from the hardware vendor. Microsoft's device drivers may not be tuned to the hardware and may not deliver the same level of performance as the hardware manufacturer's drivers. <h3>Double-check processor scheduling</h3> The Windows operating system allows for two processor scheduling models to optimize the CPU either for running programs or running background services. The correct processor scheduling model varies based on the server's workload, but a general rule is to optimize the server for background processes if the server is not equipped with a GUI. You can determine the current processor scheduling model by opening an elevated PowerShell session and running this command: <pre>Get-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\PriorityControl -Name Win32PrioritySeparation</pre> If the command returns a value of 2, then the system is optimized for programs. If PowerShell returns a value of 18, then the CPU is optimized for background services. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/posey_server_performance_improvement.jpg"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/posey_server_performance_improvement_mobile.jpg" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/posey_server_performance_improvement_mobile.jpg 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/posey_server_performance_improvement.jpg 1280w" alt="processor scheduling model" height="211" width="560"> <figcaption> The PowerShell command returns a value of 2 to indicate the processor scheduling model is set for running programs, not background services. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> The command returned a value of 2, indicating that the server is optimized for programs. To change the optimization model, use the following PowerShell command: <pre>Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\PriorityControl -Name Win32PrioritySeparation -Value <numerical value></pre> In the command above, replace <numerical value> with either 2 or 18. A small investment in time to execute these Windows Server performance tuning tips and techniques can optimize server workloads for more satisfactory results. https://cdn.ttgtmedia.com/visuals/searchBusinessAnalytics/data_analytics/businessanalytics_article_013.jpg https://www.techtarget.com/searchwindowsserver/tip/Improve-your-Windows-servers-performance-10-tips-in-10-minutes Fri, 20 Nov 2020 17:00:00 GMT Boost Windows Server performance with these 10 tips Windows administrators must maintain constant vigilance over their systems to prevent a vulnerability from crippling their systems or exposing data to threat actors. For shops that use Hyper-V, Microsoft offers another layer of protection through its virtualization-based security. Virtualization-based security uses Hyper-V and the machine's hardware virtualization features to isolate and protect an area of system memory that runs the most sensitive and critical parts of the <a href="https://www.techtarget.com/searchwindowsserver/opinion/How-to-compare-Windows-and-Linux-for-enterprise-workloads">OS kernel</a> and user modes. Once deployed, these protected areas can guard other kernel and user-mode instances. Virtualization-based security effectively reduces the Windows attack surface, so even if a malicious actor <a href="https://www.techtarget.com/searchenterprisedesktop/tip/Understanding-Windows-kernel-structure-and-why-it-matters">gains access to the OS kernel</a>, the protected content can prevent code execution and the access of secrets, such as system credentials. In theory, these added protections would prevent malware attacks that use kernel exploits from gaining access to sensitive information. <section class="section main-article-chapter" data-menu-title="Code examining, malware prevention among key capabilities"> <h2 class="section-title">Code examining, malware prevention among key capabilities</h2> Virtualization-based security is a foundation technology and must be in place before adopting a range of <a href="https://www.techtarget.com/searchwindowsserver/tip/Using-Azure-AD-conditional-access-for-tighter-security">advanced security features</a> in Windows Server. One example is Hypervisor-Enforced Code Integrity (HVCI), which examines code -- such as drivers -- and ensures the kernel mode drivers and binaries are signed before they load into memory. Unsigned content gets denied, reducing the possibility of running malicious code. Other advanced security capabilities that rely on virtualization-based security include Windows Defender Credential Guard, which prevents malware from accessing credentials, and the ability to create virtual trusted platform modules (TPMs) for shielded VMs. In Windows Server 2019, Microsoft expanded its shielded VMs feature beyond the Windows platform to cover Linux workloads running on Hyper-V to prevent data leakage when the VM is both static and when it moves to another Hyper-V host. New in Windows Server 2019 is a feature called host key attestation, which uses asymmetric key pairs to authenticate hosts covered by the Host Guardian Service in what is described as an easier deployment method by not requiring an Active Directory trust arrangement. </section> <section class="section main-article-chapter" data-menu-title="What are the virtualization-based security requirements?"> <h2 class="section-title">What are the virtualization-based security requirements?</h2> Virtualization-based security has numerous requirements. It's important to investigate the complete set of hardware, firmware and software requirements before adopting virtualization-based security. Any missing requirements may make it impossible to enable virtualization-based security and compromise system security features that depend on virtualization-based security support. At the hardware level, virtualization-based security needs a 64-bit processor with virtualization extensions (Intel VT-x and AMD-V) and <a href="https://www.techtarget.com/whatis/definition/second-level-address-translation-SLAT">second-level address translation</a> as Extended Page Tables or Rapid Virtualization Indexing. I/O virtualization must be supported through Intel VT-d or AMD-Vi. The server hardware must include TPM 2.0 or better. System firmware must support the Windows System Management Mode Security Mitigations Table specification. Unified Extensible Firmware Interface must support memory reporting features such as the UEFI v2.6 Memory Attributes Table. Support for Secure Memory Overwrite Request v2 will inhibit in-memory attacks. All drivers must be compatible with <a href="https://docs.microsoft.com/en-us/windows/security/threat-protection/device-guard/enable-virtualization-based-protection-of-code-integrity" target="_blank" rel="noopener">HVCI standards</a>. </section> Microsoft Hyper-V on Windows comes with advanced protection schemes, including several virtualization-based security features the company introduced in Windows Server 2019. https://cdn.ttgtmedia.com/visuals/searchCIO/enterprise_security/cio_article_014.jpg https://www.techtarget.com/searchwindowsserver/answer/What-are-Windows-virtualization-based-security-features Mon, 17 Feb 2020 10:30:00 GMT What are Windows virtualization-based security features? If you're considering moving your on-premises SQL Server databases to the Microsoft <a href="https://www.techtarget.com/searchbusinessanalytics/news/252473347/Microsoft-launches-Azure-Synapse-Analytics-for-the-cloud">Azure</a> cloud, you need to decide which of the different cloud options works best for you. You might not realize it to begin with, but there are three ways to implement SQL Server in Azure. You can run your SQL Server cloud workloads in an <a href="https://www.techtarget.com/searchcloudcomputing/definition/Infrastructure-as-a-Service-IaaS">IaaS</a> deployment or a hosted <a href="https://www.techtarget.com/searchcloudcomputing/definition/Platform-as-a-Service-PaaS">PaaS</a> setup. With the former, you install and run SQL Server on an Azure virtual machine (VM); with the latter, you migrate your databases to Microsoft's Azure SQL Database cloud service. Additionally, there's the newer Azure SQL Database managed instance platform, which offers something of a middle ground between the IaaS and PaaS approaches. Let's take a closer look at the three options for <a href="https://www.techtarget.com/searchcloudcomputing/news/252462956/Microsoft-expands-Azure-cloud-database-options">deploying SQL Server databases in Azure</a> and their key features, pricing models and database migration tools. <h3>SQL Server on Azure VMs</h3> The most straightforward way to deploy SQL Server in Azure is to put it on an Azure VM. To do so, you must have an Azure account. The step is to create and <a href="https://www.techtarget.com/searchcloudcomputing/tip/Map-the-best-Azure-VM-sizes-and-types-to-your-cloud-workloads">size an Azure VM</a>. Next, you'll need to install an OS and SQL Server on the VM and then migrate your databases to it. To speed up this process, <a href="https://www.techtarget.com/searchcloudcomputing/definition/Microsoft-Azure-Marketplace">Azure Marketplace</a> has several VM images with SQL Server and either Windows Server or Linux preinstalled. For example, the Windows-based SQL Server 2017 image is highlighted in the Azure portal screen shown in Figure 1. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/editorial/072519_sSQL_SQLServer2017-VM-image_Fig1.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/editorial/072519_sSQL_SQLServer2017-VM-image_Fig1_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/editorial/072519_sSQL_SQLServer2017-VM-image_Fig1_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/editorial/072519_sSQL_SQLServer2017-VM-image_Fig1.png 1280w" alt="SQL Server 2017 VM image on Azure Marketplace" height="264" width="560"> <figcaption> Figure 1. Installing a SQL Server 2017 VM image from Azure Marketplace </figcaption> <div class="main-article-image-enlarge"> </div> </figure> Selecting this option enables you to quickly create a SQL Server 2017 VM running on Windows Server 2016 without needing to manually install either the OS or the database software. There are also images for SQL Server 2017 on Linux and the <a href="https://searchsqlserver.techtarget.com/tip/SQL-Server-2019-improves-Linux-container-support">preview release of SQL Server 2019</a>, as well as earlier versions of the database management system (DBMS). Azure VMs offer up to 99.95% availability, and you have full control over the VM's configuration and the SQL Server instance. Because you're deploying your own SQL Server instance, there's complete feature parity with the on-premises version of SQL Server -- they're essentially the same thing. You're also able to deploy other applications or services on the VM host that's running SQL Server, and you can provide your own private IP addresses to the VM using Azure Virtual Network (VNet). While SQL Server systems running on Azure VMs provide the same degree of control as on-premises ones, they also have the same type of operational and management requirements. You don't have to – and, in fact, can't -- manage the underlying hardware, but you do need to manage database backups and software updates. You also need to implement whatever high availability (HA) and disaster recovery tools your SQL Server instance requires, and you're responsible for reconfiguring system resources, like CPU and storage, in an Azure VM, which will result in database downtime. You can manage SQL Server VMs in Azure using SQL Server Management Studio (<a href="https://searchsqlserver.techtarget.com/definition/Microsoft-SQL-Server-Management-Studio-SSMS">SSMS</a>), the more lightweight <a href="https://searchsqlserver.techtarget.com/definition/SQL-Operations-Studio">Azure Data Studio</a> tool, SQL Server Data Tools (SSDT) in Visual Studio or the Visual Studio Code editor with the mssql extension. Azure VMs are priced under a pay-as-you-go model that charges by the second for the use of compute capacity or an Azure Reserved VM Instances option that lets users make an advance purchase of a VM for one year or three years at a reduced capacity rate. VMs are available in a number of sizing categories that are designed for different budgets and workloads. Migration from on-premises SQL Server to Azure VMs is relatively easy and can be accomplished in a variety of ways: <ul class="default-list"> <li>copying a database backup to an Azure VM;</li> <li>using Microsoft's Backup to URL feature, in which a database is backed up in Azure Blob storage and then restored to a VM;</li> <li>detaching the data and log files, copying them to Azure Blob storage and then attaching them to a VM;</li> <li>converting an on-premises physical system or VM to a Hyper-V virtual hard disk, uploading the VHD to Azure Blob storage and then deploying it on a Azure VM;</li> <li>using an Always On availability group to create a replica database in an Azure VM; and</li> <li>running Data Migration Assistant (DMA) to migrate a database to an Azure VM.</li> </ul> <div class="youtube-iframe-container"> <iframe id="ytplayer-0" src="https://www.youtube.com/embed/T8ngx84oHFY?autoplay=0&modestbranding=1&rel=0&widget_referrer=null&enablejsapi=1&origin=https://searchsqlserver.techtarget.com" type="text/html" height="360" width="640" frameborder="0"></iframe> </div> <h3>Azure SQL Database</h3> Azure SQL Database, Microsoft's PaaS option for SQL Server users in the cloud, is more specifically a database as a service (DBaaS) offering. With Azure SQL Database, you needn't be concerned with configuring the underlying system, creating VMs or installing an OS and the database software itself. Instead, Microsoft handles all of the platform management and is responsible for security, availability, and backups. One of the big advantages of the PaaS approach is that you can get cost savings on both capital and operating expenses. Azure SQL Database also takes a lot of the heavy lifting out of a SQL Server in Azure implementation. Microsoft provides database monitoring and automatic <a href="https://searchsqlserver.techtarget.com/tip/SQL-Server-performance-tuning-best-practices-for-DBAs">performance tuning</a>, and the cloud service automatically does full, differential and transaction log backups in databases and supports point-in-time recovery. Azure SQL Database guarantees 99.99% availability and uses the latest stable version of SQL Server Database Engine, although Microsoft's default <a href="https://www.sqlskills.com/blogs/glenn/database-compatibility-level-in-sql-server/" target="_blank" rel="noopener">compatibility level settings</a> can be used to align existing databases with older versions of the DBMS if you want to do so. You can implement Azure SQL Database as a single database or an elastic pool. As its name implies, a single database consists of just one database with its own set of resources -- similar to using an on-premises contained database. An elastic pool is a collection of databases with a shared set of resources. Elastic pools are designed for managing multiple databases with usage levels that vary in unpredictable ways -- they make sure that the databases get needed processing resources when demand increases. You can scale up both single databases and elastic pools with no interruption of service. Both also enable you to configure up to four readable secondary databases to use as a failover group for HA and load balancing. As in SQL Server on Azure VMs, you can use Azure VNet to assign private IP addresses to Azure SQL Database systems. For database development and management in Azure SQL Database, you can use the Azure portal in addition to SSMS, Azure Data Studio, SSDT and Visual Studio Code. The Azure portal screen for creating a database is shown in Figure 2. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/editorial/072519_sSQL_Azure-portal-screen_Fig2.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/editorial/072519_sSQL_Azure-portal-screen_Fig2_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/editorial/072519_sSQL_Azure-portal-screen_Fig2_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/editorial/072519_sSQL_Azure-portal-screen_Fig2.png 1280w" alt="Azure portal screen for creating a database in Azure SQL Database" height="373" width="559"> <figcaption> Figure 2. Creating a database in the Azure SQL Database cloud service </figcaption> <div class="main-article-image-enlarge"> </div> </figure> Azure SQL Database has a high level of compatibility with on-premises SQL Server. However, a number of on-premises features aren't supported in the cloud service. For example, you can't attach a database to an Azure SQL Database instance or run manual backup commands. Change data capture, <a href="https://www.techtarget.com/whatis/definition/Common-Language-Runtime-CLR">Common Language Runtime extensions</a>, Database Mail, database snapshots, distributed partitioned views, linked servers and SQL Server Agent are some of the other features that aren't supported. Microsoft offers two pricing models for Azure SQL Database. The original one is a pay-as-you-go model based on Database Transaction Units (DTUs), a resource utilization metric created by Microsoft. The DTU-based model includes three service tiers to support light to heavy database workloads. Different system sizes within each tier provide a preconfigured mix of compute, and storage resources. In 2018, Microsoft added a vCore-based pricing model that allows you to choose the number of virtual processor cores and the amount of and storage needed to run your database workloads. Microsoft now recommends use of the vCore model, saying it offers a more straightforward way to translate on-premises SQL Server workload requirements to Azure SQL Database. Under the vCore approach, customers with existing SQL Server licenses and Software Assurance maintenance contracts can also use the Azure Hybrid Benefit pricing discount to get cost savings; additionally, a reserved capacity option enables companies to prepay for vCores for one year or three years at a lower rate. The recommended options for migrating on-premises SQL Server databases to Azure SQL Database are more limited than the ones for moving to SQL Server on an Azure VM. You can use DMA, either separately or as part of Azure Database Migration Service, or you can use transactional replication, in which Azure SQL Database is set up to act as a subscriber to an on-premises SQL Server instance that continuously publishes changes to database tables. <h3>Azure SQL Database managed instance</h3> The Azure SQL Database managed instance deployment option became available in 2018. It's a fully managed DBaaS offering that gives SQL Server in Azure users all the PaaS benefits of Azure SQL Database. However, it also includes most of the additional on-premises capabilities that were previously available to cloud users only in SQL Server on Azure VMs. Azure SQL Database managed instance provides near-100% compatibility with on-premises SQL Server, according to Microsoft; a managed instance in Azure is similar to an on-premises instance. There are still some differences in functionality, though -- for example, in backups, HA and security. Also, tools like SQL Server Analysis Services, Integration Services and Reporting Services aren't built in, although their functions can be provided through other Azure services. There are two different service tiers for Azure SQL Database managed instance -- a General Purpose one for applications with typical workloads and a Business Critical tier for workloads that require high performance and low I/O latency. A managed instance can host up to 100 databases on as much as 8 TB of storage in the General Purpose tier and 4 TB in the Business Critical one, depending on the selected performance level and the number of vCores configured in a system. Like the single database and elastic pool versions of Azure SQL Database, the managed instance offering provides 99.99% availability and supports private IP addresses through Azure VNet The primary goal of Azure SQL Database managed instance is to make it easier to migrate your SQL Server instances to Azure. Currently, that could be a particularly appealing option to companies still running SQL Server 2008 and SQL Server 2008 R2. Microsoft <a href="https://searchsqlserver.techtarget.com/tip/4-strategies-to-beat-the-SQL-Server-2008-support-cutoff">ended support</a> for those two versions of the DBMS on July 9, 2019, and many of the remaining installations run on aging hardware platforms. Azure SQL Database managed instance supports migration from SQL Server 2008 and 2008 R2, as well as newer SQL Server versions, with minimal or no database changes. As shown in Figure 3, you can create a new managed instance in the Azure portal by selecting Create a resource, then Databases, followed by Azure SQL Managed Instance. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/editorial/072519_sSQL_Azure-SQL-Database-managed-instance_Fig3.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/editorial/072519_sSQL_Azure-SQL-Database-managed-instance_Fig3_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/editorial/072519_sSQL_Azure-SQL-Database-managed-instance_Fig3_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/editorial/072519_sSQL_Azure-SQL-Database-managed-instance_Fig3.png 1280w" alt="Azure portal screen for creating an Azure SQL Database managed instance" height="322" width="560"> <figcaption> Figure 3. Creating an Azure SQL Database managed instance </figcaption> <div class="main-article-image-enlarge"> </div> </figure> Azure SQL Database managed instance uses a vCore-based pricing model with two compute levels built around different processors, one supporting eight to 24 vCores and the other four to 80. As with the other Azure SQL Database deployment options, you can take advantage of the Azure Hybrid Benefit discount if you have Software Assurance with existing SQL Server licenses. You can also pay upfront for reserved capacity at a discount rate. Microsoft supports two methods of migrating databases to Azure SQL Database managed instance for users looking to run SQL Server in Azure. You can restore a database from a native SQL Server backup that you copy to Azure storage, or you can use Azure Database Migration Service with DMA. SQL Server databases can be moved to the Azure cloud in several different ways. Here's what you'll get from each of the options for migrating SQL Server to Azure. https://cdn.ttgtmedia.com/visuals/searchCloudApplications/paas/cloudapplications_article_011.jpg https://www.techtarget.com/searchdatamanagement/tip/SQL-Server-in-Azure-database-choices-and-what-they-offer-users Mon, 29 Jul 2019 16:41:00 GMT SQL Server in Azure database choices and what they offer users In this excerpt from SQL Queries for Mere Mortals: A Hands-On Guide to Data Manipulation in SQL, John L. Viescas and Michael J. Hernandez discuss the LEFT OUTER JOIN vs. the RIGHT OUTER JOIN and offer tips on how to use each of them in SQL statements. They also provide various examples of OUTER JOIN syntax to help guide database administrators and developers on the required SQL coding. You'll generally use the OUTER JOIN form that asks for all the rows from one table or result set and any matching rows from a second table or result set. To do this, you specify either a LEFT OUTER JOIN or a RIGHT OUTER JOIN. What's the <a href="https://www.techtarget.com/searchoracle/answer/Oracle-LEFT-JOIN-vs-LEFT-OUTER-JOIN-Whats-the-difference">difference between LEFT and RIGHT</a>? Remember from the previous chapter that to specify an INNER JOIN on two tables, you name the first table, include the JOIN keyword, and then name the second table. When you begin <a href="http://technet.microsoft.com/en-us/library/ms191517(v=sql.105).aspx">building queries</a> using OUTER JOIN, the SQL Standard considers the first table you name as the one on the "left," and the second table as the one on the "right." So, if you want all the rows from the first table and any matching rows from the second table, you'll use a <a href="https://www.techtarget.com/searchoracle/answer/What-is-the-difference-between-left-outer-join-and-outer-join">LEFT OUTER JOIN</a>. Conversely, if you want all the rows from the second table and any matching rows from the first table, you'll specify a RIGHT OUTER JOIN. <h3>Syntax</h3> Let's examine the <a href="https://www.techtarget.com/whatis/definition/syntax">syntax</a> needed to build either a LEFT or RIGHT OUTER JOIN. <h3>Using Tables</h3> We'll start simply with <a href="https://www.techtarget.com/searchoracle/answer/LEFT-OUTER-JOIN-without-using-LEFT-OUTER-JOIN">defining an OUTER JOIN</a> using tables. Figure 9–3 shows the syntax diagram for creating a query with an OUTER JOIN on two tables. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.3.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.3_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.3_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.3.png 1280w" height="333" width="560"> <figcaption> Figure 9–3 Defining an OUTER JOIN on two tables. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> Just like INNER JOIN (covered in Chapter 8), all the action happens in the FROM clause. (We left out the WHERE and ORDER BY clauses for now to simplify things.) Instead of specifying a single table name, you specify two table names and link them with the JOIN keyword. If you do not specify the type of JOIN you want, your database system assumes you want an INNER JOIN. In this case, because you want an OUTER JOIN, you must explicitly state that you want either a LEFT JOIN or a RIGHT JOIN. The OUTER keyword is optional. <div class="extra-info"> <div class="extra-info-inner"> <h4>Note</h4> For those of you following along with the complete syntax diagrams in Appendix A, SQL Standard Diagrams, note that we've pulled together the applicable parts (from Select Statement, Table Reference, and Joined Table) into simpler diagrams that explain the specific syntax we're discussing. </div> </div> The critical part of any JOIN is the ON or USING clause that follows the second table and tells your database system how to perform the JOIN. To solve the JOIN, your database system logically combines every row in the first table with every row in the second table. (This combination of all rows from one table with all rows from a second table is called a Cartesian product.) It then applies the criteria in the ON or USING clause to find the matching rows to be returned. Because you asked for an OUTER JOIN, your database system also returns the unmatched rows from either the "left" or "right" table. You learned about using a search condition to form a WHERE clause in Chapter 6, Filtering Your Data. You can use a search condition in the ON clause within a JOIN to specify a logical test that must be true in order to return any two linked rows. It only makes sense to write a search condition that compares at least one column from the first table with at least one column from the second table. Although you can write a very complex search condition, you can usually specify a simple equals comparison test on the primary key columns from one table with the foreign key columns from the other table. To keep things simple, let's start with the same recipe classes and recipes example we used in the last chapter. Remember that in a well-designed database, you should break out complex classification names into a second table and then link the names back to the primary subject table via a simple key value. In the Recipes sample database, recipe classes appear in a table separate from recipes. Figure 9–4 shows the relationship between the Recipe_Classes and Recipes tables. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.4.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.4_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.4_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.4.png 1280w" height="195" width="558"> <figcaption> Figure 9–4 Recipe classes are in a separate table from recipes. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> When you originally set up the kinds of recipes to save in your database, you might have started by entering all the recipe classes that came to mind. Now that you've entered a number of recipes, you might be interested in finding out which classes don't have any recipes entered yet. You might also be interested in listing all the recipe classes along with the names of recipes entered so far for each class. You can solve either problem with an OUTER JOIN. <div class="extra-info"> <div class="extra-info-inner"> <h4>Note</h4> Throughout this chapter, we use the "Request/Translation/Clean Up/SQL" technique introduced in Chapter 4, Creating a Simple Query. </div> </div> "Show me all the recipe types and any matching recipes in my database." <table cellspacing="03" border="1" cellpadding="03" style="width: 100%;"> <tbody> <tr> <td width="20%">Translation</td> <td width="80%">Select recipe class description and recipe title from the recipe classes table left outer joined with the recipes table on recipe class ID in the recipe classes table matching recipe class ID in the recipes table</td> </tr> <tr> <td width="20%">Clean Up</td> <td width="80%">Select recipe class description <s>and </s>recipe title from <s>the</s>recipe classes <s>table</s> left outer join<s>ed with the</s> recipes <s>table</s> on recipe_ classes.recipe class ID <s>in the recipe classes table</s> <s>matching</s> = recipes.recipe class ID <s>in the recipes table</s></td> </tr> <tr> <td width="20%">SQL</td> <td width="80%"><code>SELECT Recipe_Classes.RecipeClassDescription,</code> <code>Recipes.RecipeTitle </code> <code>FROM Recipe_Classes </code> <code>LEFT OUTER JOIN Recipes </code> <code>ON Recipe_Classes.RecipeClassID = </code> <code>Recipes.RecipeClassID</code></td> </tr> </tbody> </table> When using multiple tables in your FROM clause, remember to qualify fully each column name with the table name wherever you use it so that it's absolutely clear which column from which table you want. Note that we had to qualify the name of RecipeClassID in the ON clause because there are two columns named RecipeClassID -- one in the Recipes table and one in the Recipe_Classes table. <div class="extra-info"> <div class="extra-info-inner"> <h4>Note</h4> Although most commercial implementations of SQL support OUTER JOIN, some do not. If your database does not support OUTER JOIN, you can still solve the problem by listing all the tables you need in the FROM clause, then moving your search condition from the ON clause to the WHERE clause. You must consult your database documentation to learn the specific nonstandard syntax that your database requires to define the OUTER JOIN. For example, earlier versions of Microsoft SQL Server support this syntax. (Notice the asterisk in the WHERE clause.) SELECT Recipe_Classes.RecipeClassDescription, Recipes.RecipeTitle FROM Recipe_Classes, Recipes WHERE Recipe_Classes.RecipeClassID *= Recipes.RecipeClassID If you're using Oracle, the optional syntax is as follows. (Notice the plus sign in the WHERE clause.) SELECT Recipe_Classes.RecipeClassDescription, Recipes.RecipeTitle FROM Recipe_Classes, Recipes WHERE Recipe_Classes.RecipeClassID = Recipes.RecipeClassID(+) Quite frankly, these strange syntaxes were invented by database vendors that wanted to provide this feature long before a clearer syntax was defined in the SQL Standard. Thankfully, the SQL Standard syntax allows you to fully define the source for the final result set entirely within the FROM clause. Think of the FROM clause as fully defining a linked result set from which the database system obtains your answer. In the SQL Standard, you use the WHERE clause only to filter rows out of the result set defined by the FROM clause. Also, because the specific syntax for defining an OUTER JOIN via the WHERE clause varies by product, you might have to learn several different syntaxes if you work with multiple nonstandard products. </div> </div> If you execute our example query in the Recipes sample database, you should see 16 rows returned. Because we didn't enter any soup recipes in the database, you'll get a Null value for RecipeTitle in the row where RecipeClassDescription is 'Soup'. To find only this one row, use this approach. "List the recipe classes that do not yet have any recipes." <table cellspacing="03" border="1" cellpadding="03" style="width: 100%;"> <tbody> <tr> <td width="20%">Translation</td> <td width="80%">Select recipe class description from the recipe classes table left outer joined with the recipes table on recipe class ID where recipe ID is empty</td> </tr> <tr> <td width="20%">Clean Up</td> <td width="80%">Select recipe class description from <s>the</s>recipe classes <s>table</s> left outer join<s>ed with the</s> recipes <s>table</s> on recipe_classes.recipe class ID <s>in the recipes table matches</s> = recipes.recipe class ID <s>in the recipes table</s> where recipe ID is <s>empty</s> NULL</td> </tr> <tr> <td width="20%">SQL</td> <td width="80%"><code>SELECT Recipe_Classes.RecipeClassDescription</code> <code>FROM Recipe_Classes</code> <code>LEFT OUTER JOIN Recipes</code> <code>ON Recipe_Classes.RecipeClassID = </code> <code>Recipes.RecipeClassID</code> <code>WHERE Recipes.RecipeID IS NULL</code></td> </tr> </tbody> </table> If you think about it, we've just done a difference or EXCEPT operation (see Chapter 7) using a JOIN. It's somewhat like saying, "Show me all the recipe classes except the ones that already appear in the recipes table." The set diagram in Figure 9–5 should help you visualize what's going on. <figure class="main-article-image half-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.5.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.5_half_column_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.5_half_column_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.5.png 1280w" height="277" width="279"> <figcaption> Figure 9–5 A possible relationship between recipe classes and recipes. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> In Figure 9–5, all recipes have a recipe class, but some recipe classes exist for which no recipe has yet been defined. When we add the IS NULL test, we're asking for all the rows in the lighter outer circle that don't have any matches in the set of recipes represented by the darker inner circle. Notice that the diagram for an OUTER JOIN on tables in Figure 9–3 also has the optional USING clause. If the matching columns in the two tables have the same name and you want to join only on equal values, you can use the USING clause and list the column names. Let's do the previous problem again with USING. "Display the recipe classes that do not yet have any recipes." <table cellspacing="03" border="1" cellpadding="03" style="width: 100%;"> <tbody> <tr> <td width="20%">Translation</td> <td width="80%">Select recipe class description from the recipe classes table left outer joined with the recipes table using recipe class ID where recipe ID is empty</td> </tr> <tr> <td width="20%">Clean Up</td> <td width="80%">Select recipe class description from <s>the</s>recipe classes <s>table</s> left outer join<s>ed with the</s> recipes <s>table</s> using recipe class ID where recipe ID is <s>empty</s> NULL</td> </tr> <tr> <td width="20%">SQL</td> <td width="80%"><code>SELECT Recipe_Classes.RecipeClassDescription</code> <code>FROM Recipe_Classes</code> <code>LEFT OUTER JOIN Recipes</code> <code> USING (RecipeClassID)</code> <code>WHERE Recipes.RecipeID IS NULL</code></td> </tr> </tbody> </table> The USING syntax is a lot simpler, isn't it? There's one small catch: Any column in the USING clause loses its table identity because the SQL Standard dictates that the database system must "coalesce" the two columns into a single column. In this example, there's only one RecipeClassID column as a result, so you can't reference Recipes.RecipeClassID or Recipe_Classes.RecipeClassID in the SELECT clause or any other clause. Be aware that some database systems do not yet support USING. If you find that you can't use USING with your database, you can always get the same result with an ON clause and an equals comparison. <div class="extra-info"> <div class="extra-info-inner"> <h4>Note</h4> The SQL Standard also defines a type of JOIN operation called a NATURAL JOIN. A NATURAL JOIN links the two specified tables by matching all the columns with the same name. If the only common columns are the linking columns and your database supports NATURAL JOIN, you can solve the example problem like this: SELECT Recipe_Classes.RecipeClassDescription FROM Recipe_Classes NATURAL LEFT OUTER JOIN Recipes WHERE Recipes.RecipeID IS NULL Do not specify an ON or USING clause if you use the NATURAL keyword. </div> </div> Embedding a SELECT Statement As you recall from Chapter 8, most SQL implementations let you substitute an entire SELECT statement for any table name in your FROM clause. Of course, you must then assign a correlation name (see the section on assigning alias names in Chapter 8) so that the result of evaluating your embedded query has a name. Figure 9–6 shows how to assemble an OUTER JOIN clause using embedded SELECT statements. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.6.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.6_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.6_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.6.png 1280w" height="331" width="560"> <figcaption> Figure 9–6 An OUTER JOIN using SELECT statements. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> Note that a SELECT statement can include all query clauses except an ORDER BY clause. Also, you can mix and match SELECT statements with table names on either side of the OUTER JOIN keywords. Let's look at the Recipes and Recipe_Classes tables again. For this example, let's also assume that you are interested only in classes Salads, Soups, and Main courses. Here's the query with the Recipe_Classes table filtered in a SELECT statement that participates in a LEFT OUTER JOIN with the Recipes table. SQL SELECT RCFiltered.ClassName, R.RecipeTitle FROM (SELECT RecipeClassID, RecipeClassDescription AS ClassName FROM Recipe_Classes AS RC WHERE RC.ClassName = 'Salads' OR RC.ClassName = 'Soup' OR RC.ClassName = 'Main Course') AS RCFiltered LEFT OUTER JOIN Recipes AS R ON RCFiltered.RecipeClassID = R.RecipeClassID You must be careful when using a SELECT statement in a FROM clause. First, when you decide to substitute a SELECT statement for a table name, you must be sure to include not only the columns you want to appear in the final result but also any linking columns you need to perform the JOIN. That's why you see both RecipeClassID and RecipeClassDescription in the embedded statement. Just for fun, we gave RecipeClassDescription an alias name of ClassName in the embedded statement. As a result, the SELECT clause asks for ClassName rather than RecipeClassDescription. Note that the ON clause now references the correlation name (RCFiltered) of the embedded SELECT statement rather than the original name of the table or the correlation name we assigned the table inside the embedded SELECT statement. As the query is stated for the actual Recipes sample database, you see one row with RecipeClassDescription of Soup with a Null value returned for RecipeTitle because there are no soup recipes in the sample database. We could just as easily have built a SELECT statement on the Recipes table on the right side of the OUTER JOIN. For example, we could have asked for recipes that contain the word "beef" in their titles, as in the following statement. SQL SELECT RCFiltered.ClassName, R.RecipeTitle FROM (SELECT RecipeClassID, RecipeClassDescription AS ClassName FROM Recipe_Classes AS RC WHERE RC.ClassName = 'Salads' OR RC.ClassName = 'Soup' OR RC.ClassName = 'Main Course') AS RCFiltered LEFT OUTER JOIN (SELECT Recipes.RecipeClassID, Recipes.Recipe Title FROM Recipes WHERE Recipes.RecipeTitle LIKE '%beef%') AS R ON RCFiltered.RecipeClassID = R.RecipeClassID Keep in mind that the LEFT OUTER JOIN asks for all rows from the result set or table on the left side of the JOIN, regardless of whether any matching rows exist on the right side. The previous query not only returns a Soup row with a Null RecipeTitle (because there are no soups in the database at all) but also a Salad row with a Null. You might conclude that there are no salad recipes in the database. Actually, there are salads in the database but no salads with "beef" in the title of the recipe! <div class="extra-info"> <div class="extra-info-inner"> <h4>Note</h4> You might have noticed that you can enter a full search condition as part of the ON clause in a JOIN. This is absolutely true, so it is perfectly legal in the SQL Standard to solve the example problem as follows. SELECT Recipe_Classes.RecipeClassDescription, Recipes.RecipeTitle FROM Recipe_Classes LEFT OUTER JOIN Recipes ON Recipe_Classes.RecipeClassID = Recipes.RecipeClassID AND (Recipe_Classes.RecipeClassDescription = 'Salads' OR Recipe_Classes.RecipeClassDescription = 'Soup' OR Recipe_Classes.RecipeClassDescription = 'Main Course') AND Recipes.RecipeTitle LIKE '%beef%' Unfortunately, we have discovered that some major implementations of SQL solve this problem incorrectly or do not accept this syntax at all! Therefore, we recommend that you always enter in the search condition in the ON clause only criteria that compare columns from the two tables or result sets. If you want to filter the rows from the underlying tables, do so with a separate search condition in a WHERE clause in an embedded SELECT statement. </div> </div> <h3>Embedding JOINs within JOINs</h3> Although you can solve many problems by linking just two tables, many times you'll need to link three, four, or more tables to get all the data to solve your request. For example, you might want to fetch all the relevant information about recipes -- the type of recipe, the recipe name, and all the ingredients for the recipe -- in one query. Now that you understand what you can do with an OUTER JOIN, you might also want to list all recipe classes -- even those that have no recipes defined yet -- and all the details about recipes and their ingredients. Figure 9–7 shows all the tables needed to answer this request. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.7.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.7_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.7_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.7.png 1280w" height="224" width="560"> <figcaption> Figure 9–7 The tables you need from the Recipes sample database to fetch all the information about recipes. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> Looks like you need data from five different tables! Just as in Chapter 8, you can do this by constructing a more complex FROM clause, embedding JOIN clauses within JOIN clauses. <a href="http://msdn.microsoft.com/en-us/library/ms173815.aspx">Here's the trick</a>: Everywhere you can specify a table name, you can also specify an entire JOIN clause surrounded with parentheses. Figure 9–8 shows a simplified version of joining two tables. (We've left off the correlation name clauses and chosen the ON clause to form a simple INNER or OUTER JOIN of two tables.) <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.8.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.8_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.8_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.8.png 1280w" height="260" width="560"> <figcaption> Figure 9–8 A simple JOIN of two tables. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> To add a third table to the mix, just place an open parenthesis before the first table name, add a close parenthesis after the search condition, and then insert another JOIN, a table name, the ON keyword, and another search condition. Figure 9–9 (on page 306) shows how to do this. If you think about it, the JOIN of two tables inside the parentheses forms a logical table, or inner result set. This result set now takes the place of the first simple table name in Figure 9–8. You can continue this process of enclosing an entire JOIN clause in parentheses and then adding another JOIN keyword, table name, ON keyword, and search condition until you have all the result sets you need. Let's make a request that needs data from all the tables shown in Figure 9–7 and see how it turns out. (You might use this type of request for a report that lists all recipe types with details about the recipes in each type.) <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.9.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.9_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.9_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.9.png 1280w" height="323" width="560"> <figcaption> Figure 9–9 A simple JOIN of three tables. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> "I need all the recipe types, and then the matching recipe names, preparation instructions, ingredient names, ingredient step numbers, ingredient quantities, and ingredient measurements from my recipes database, sorted in recipe title and step number sequence." <table cellspacing="03" border="1" cellpadding="03" style="width: 100%;"> <tbody> <tr> <td width="20%">Translation</td> <td width="80%">Select the recipe class description, recipe title, preparation instructions, ingredient name, recipe sequence number, amount, and measurement description from the recipe classes table left outer joined with the recipes table on recipe class ID in the recipe classes table matching recipe ID in the recipes table, then joined with the recipe ingredients table on recipe ID in the recipes table matching recipe ID in the recipe ingredients table, then joined with the ingredients table on ingredient ID in the ingredients table matching ingredient ID in the recipe ingredients table, and then finally joined with the measurements table on measurement amount ID in the measurements table matching measurement amount ID in the recipe ingredients table, order by recipe title and recipe sequence number</td> </tr> <tr> <td width="20%">Clean Up</td> <td width="80%">Select <s>the</s> recipe class description, recipe title, preparation <s>instructions</s>, ingredient name, recipe sequence number, amount, <s>and</s> measurement description from <s>the</s>recipe classes <s>table</s> left outer join<s>ed with the</s> recipes <s>table</s> on recipe_classes .recipe class ID <s>in the recipe classes table</s> <s>matching</s> = recipes .recipe class ID <s>in the recipes table,</s> <s>then</s> inner join<s>ed with the</s> recipe ingredients <s>table</s> on recipes.recipe ID <s>in the recipes table matching</s> = recipe_ingredients.recipe ID <s>in the recipe ingredients table, then</s> inner join<s>ed with the</s> ingredients <s>table</s> on ingredients.ingredient ID <s>in the ingredients table matching</s> = recipe_ingredients.ingredient ID <s>in the recipe ingredients table, and then</s> <s>finally</s> inner join<s>ed with the</s> measurements <s>table</s> on measurements.measurement amount ID <s>in the</s> <s>measurements table matching</s>= recipe_ingredients.measurement amount ID <s>in the recipe ingredients table,</s> order by recipe title, <s>and</s> recipe sequence number</td> </tr> <tr> <td width="20%">SQL</td> <td width="80%"><code>SELECT Recipe_Classes.RecipeClassDescription,</code> <code>Recipes.RecipeTitle, Recipes.Preparation,</code> <code>Ingredients.IngredientName,</code> <code>Recipe_Ingredients.RecipeSeqNo,</code> <code>Recipe_Ingredients.Amount,</code> <code>Measurements.MeasurementDescription</code> <code>FROM (((Recipe_Classes</code> <code>LEFT OUTER JOIN Recipes</code> <code>ON Recipe_Classes.RecipeClassID =</code> <code>Recipes.RecipeClassID)</code> <code>INNER JOIN Recipe_Ingredients</code> <code>ON Recipes.RecipeID = </code> <code>Recipe_Ingredients.RecipeID)</code> <code>INNER JOIN Ingredients</code> <code>ON Ingredients.IngredientID =</code> <code>Recipe_Ingredients.IngredientID)</code> <code>INNER JOIN Measurements </code> <code>ON Measurements.MeasureAmountID = </code> <code>Recipe_Ingredients.MeasureAmountID</code> <code>ORDER BY RecipeTitle, RecipeSeqNo</code></td> </tr> </tbody> </table> In truth, you can substitute an entire JOIN of two tables anywhere you might otherwise place only a table name. In Figure 9–9, we implied that you must first join the first table with the second table and then join that result with the third table. You could also join the second and third tables first (as long as the third table is, in fact, related to the second table and not the first one) and then perform the final JOIN with the first table. Figure 9–10 (on page 308) shows you this alternate method. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.10.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.10_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.10_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.10.png 1280w" height="333" width="560"> <figcaption> Figure 9–10 Joining more than two tables in an alternate sequence. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> To solve the request we just showed you using five tables, we could have also stated the SQL as follows. SQL SELECT Recipe_Classes.RecipeClassDescription,  Recipes.RecipeTitle, Recipes.Preparation,   Ingredients.IngredientName,   Recipe_Ingredients.RecipeSeqNo,   Recipe_Ingredients.Amount,   Measurements.MeasurementDescription  FROM Recipe_Classes   LEFT OUTER JOIN   (((Recipes   INNER JOIN Recipe_Ingredients   ON Recipes.RecipeID = Recipe_Ingredients.RecipeID)   INNER JOIN Ingredients   ON Ingredients.IngredientID =   Recipe_Ingredients.IngredientID)   INNER JOIN Measurements   ON Measurements.MeasureAmountID =  Recipe_Ingredients.MeasureAmountID)   ON Recipe_Classes.RecipeClassID =   Recipes.RecipeClassID  ORDER BY RecipeTitle, RecipeSeqNo Remember that the optimizers in some database systems are sensitive to the sequence of the JOIN definitions. If your query with many JOINs is taking a long time to execute on a large database, it might run faster if you change the sequence of JOINs in your SQL statement. You might have noticed that we used only one OUTER JOIN in the previous multiple-JOIN examples. You're probably wondering whether it's possible or even makes sense to use more than one OUTER JOIN in a complex JOIN. Let's assume that there are not only some recipe classes that don't have matching recipe rows but also some recipes that don't have any ingredients defined yet. In the previous example, you won't see any rows from the Recipes table that do not have any matching rows in the Recipe_Ingredients table because the INNER JOIN eliminates them. Let's ask for all recipes as well. "I need all the recipe types, and then all the recipe names and preparation instructions, and then any matching ingredient names, ingredient step numbers, ingredient quantities, and ingredient measurements from my recipes database, sorted in recipe title and step number sequence." <table cellspacing="03" border="1" cellpadding="03" style="width: 100%;"> <tbody> <tr> <td width="20%">Translation</td> <td width="80%">Select the recipe class description, recipe title, preparation instructions, ingredient name, recipe sequence number, amount, and measurement description from the recipe classes table left outer joined with the recipes table on recipe class ID in the recipe classes table matching recipe class ID in the recipes table, then left outer joined with the recipe ingredients table on recipe ID in the recipes table matching recipe ID in the recipe ingredients table, then joined with the ingredients table on ingredient ID in the ingredients table matching ingredient ID in the recipe ingredients table, and then finally joined with the measurements table on measurement amount ID in the measurements table matching measurement amount ID in the recipe ingredients table, order by recipe title and recipe sequence number</td> </tr> <tr> <td width="20%">Clean Up</td> <td width="80%">Select <s>the</s> recipe class description, recipe title, preparation <s>instructions</s>, ingredient name, recipe sequence number, amount, <s>and</s> measurement description from <s>the</s>recipe classes <s>table</s> left outer join<s>ed with the</s> recipes <s>table</s> on recipe_classes.recipe class ID <s>in the recipe classes table</s> <s>matching</s> = recipes.recipe class ID <s>in the recipes table,</s> <s>then</s> left outer join<s>ed with the</s> recipe ingredients <s>table</s> on recipes.recipe ID <s>in the recipes table matching</s> = recipe_ingredients.recipe ID <s>in the recipe ingredients table, then</s> inner join<s>ed with the</s> ingredients <s>table</s> on ingredients.ingredient ID <s>in the ingredients table matching</s> = recipe_ingredients.ingredient ID <s>in the recipe ingredients table,</s> <s>and then finally</s> inner join<s>ed with the</s> measurements <s>table</s> on measurement.measurement amount ID <s>in the measurements table matching</s> = recipe_ingredients.measurement amount ID <s>in the recipe ingredients table,</s> order by recipe title <s>and</s> recipe sequence number</td> </tr> <tr> <td width="20%">SQL</td> <td width="80%"><code>SELECT Recipe_Classes.RecipeClassDescription,</code> <code>Recipes.RecipeTitle, Recipes.Preparation, </code> <code>Ingredients.IngredientName, </code> <code>Recipe_Ingredients.RecipeSeqNo, </code> <code>Recipe_Ingredients.Amount, </code> <code>Measurements.MeasurementDescription</code> <code>FROM (((Recipe_Classes </code> <code>LEFT OUTER JOIN Recipes </code> <code>ON Recipe_Classes.RecipeClassID = </code> <code>Recipes.RecipeClassID) </code> <code>LEFT OUTER JOIN Recipe_Ingredients </code> <code>ON Recipes.RecipeID = </code> <code>Recipe_Ingredients.RecipeID) </code> <code>INNER JOIN Ingredients </code> <code>ON Ingredients.IngredientID = </code> <code>Recipe_Ingredients.IngredientID) </code> <code>INNER JOIN Measurements </code> <code>ON Measurements.MeasureAmountID = </code> <code>Recipe_Ingredients.MeasureAmountID</code> <code>ORDER BY RecipeTitle, RecipeSeqNo</code></td> </tr> </tbody> </table> Be careful! This sort of multiple OUTER JOIN works as expected only if you're following a path of one-to-many relationships. Let's look at the relationships between Recipe_Classes, Recipes, and Recipe_Ingredients again, as shown in Figure 9–11. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.11.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.11_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.11_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.11.png 1280w" height="121" width="558"> <figcaption> Figure 9–11 The relationships between the Recipe_Classes, Recipes, and Recipe_Ingredients tables. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> You might see a one-to-many relationship sometimes called a parent-child relationship. Each parent row (on the "one" side of the relationship) might have zero or more children rows (on the "many" side of the relationship). Unless you have orphaned rows on the "many" side (for example, a row in Recipes that has a Null in its RecipeClassID column), every row in the child table should have a matching row in the parent table. So it makes sense to say Recipe_Classes LEFT JOIN Recipes to pick up any parent rows in Recipe_Classes that don't have any children yet in Recipes. Recipe_Classes RIGHT JOIN Recipes should (barring any orphaned rows) give you the same result as an INNER JOIN. Likewise, it makes sense to ask for Recipes LEFT JOIN Recipe_ Ingredients because you might have some recipes for which no ingredients have yet been entered. Recipes RIGHT JOIN Recipe_Ingredients doesn't work because the linking column (RecipeID) in Recipe_Ingredients is also part of that table's compound primary key. Therefore, you are guaranteed to have no orphaned rows in Recipe_Ingredients because no column in a primary key can contain a Null value. Now, let's take it one step further and ask for all ingredients, including those not yet included in any recipes. First, take a close look at the relationships between the tables, including the Ingredients table, as shown in Figure 9–12 <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.12.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.12_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.12_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-figure_9.12.png 1280w" height="215" width="560"> <figcaption> Figure 9–12 The relationships between the Recipe_Classes, Recipes, Recipe_ Ingredients, and Ingredients tables. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> Let's try this request. (Caution: There's a trap here!) "I <a id="Need" name="Need"></a> need all the recipe types, and then all the recipe names and preparation instructions, and then any matching ingredient step numbers, ingredient quantities, and ingredient measurements, and finally all ingredient names from my recipes database, sorted in recipe title and step number sequence." <table cellspacing="3" border="1" cellpadding="3" style="width: 100%;"> <tbody> <tr> <td width="20%">Translation</td> <td width="80%">Select the recipe class description, recipe title, preparation instructions, ingredient name, recipe sequence number, amount, and measurement description from the recipe classes table left outer joined with the recipes table on recipe class ID in the recipe classes table matches class ID in the recipes table, then left outer joined with the recipe ingredients table on recipe ID in the recipes table matches recipe ID in the recipe ingredients table, then joined with the measurements table on measurement amount ID in the measurements table matches measurement amount ID in the measurements table, and then finally right outer joined with the ingredients table on ingredient ID in the ingredients table matches ingredient ID in the recipe ingredients table, order by recipe title and recipe sequence number</td> </tr> <tr> <td width="20%">Clean Up</td> <td width="80%">Select <s>the</s> recipe class description, recipe title, preparation <s>instructions</s>, ingredient name, recipe sequence number, amount, <s>and</s> measurement description from <s>the</s>recipe classes <s>table</s> left outer join<s>ed with the</s> recipes <s>table</s> on recipe_classes .recipe class ID <s>in the recipe classes table matches</s> = recipes .class ID <s>in the recipes table,then</s>left outer join<s>ed with the</s> recipe ingredients <s>table</s> on recipes.recipe ID <s>in the recipes table matches</s> = recipe_ingredients.recipe ID <s>in the recipe ¬ingredients table,</s> <s>then</s> inner join<s>ed with the</s> measurements <s>table</s> on measurements.measurement amount ID <s>in the measurements table matches</s> = measurements.measurement amount ID <s>in the measurements table,</s> <s>and then finally</s> right outer join<s>ed with the</s> ingredients <s>table</s> on ingredients.ingredient ID <s>in the ingredients table matches</s> = recipe_ingredients.ingredient ID <s>in the recipe ingredients table,</s> order by recipe title, <s>and</s> recipe sequence number</td> </tr> <tr> <td width="20%">SQL</td> <td width="80%"><code>SELECT Recipe_Classes.RecipeClassDescription,</code> <code>Recipes.RecipeTitle, Recipes.Preparation, </code> <code> Ingredients.IngredientName, </code> <code>Recipe_Ingredients.RecipeSeqNo, </code> <code>Recipe_Ingredients.Amount, </code> <code>Measurements.MeasurementDescription</code> <code>FROM (((Recipe_Classes </code> <code>LEFT OUTER JOIN Recipes </code> <code>ON Recipe_Classes.RecipeClassID = </code> <code>Recipes.RecipeClassID) </code> <code>LEFT OUTER JOIN Recipe_Ingredients </code> <code>ON Recipes.RecipeID = </code> <code>Recipe_Ingredients.RecipeID) </code> <code>INNER JOIN Measurements </code> <code>ON Measurements.MeasureAmountID =</code> <code>Recipe_Ingredients.MeasureAmountID) </code> <code>RIGHT OUTER JOIN Ingredients </code> <code>ON Ingredients.IngredientID = </code> <code>Recipe_Ingredients.IngredientID </code> <code>ORDER BY RecipeTitle, RecipeSeqNo</code></td> </tr> </tbody> </table> Do you think this will work? Actually, the answer is a resounding NO! Most database systems analyze the entire FROM clause and then try to determine the most efficient way to assemble the table links. Let's assume, however, that the database decides to fully honor how we've grouped the JOINs within parentheses. This means that the database system will work from the innermost JOIN first (Recipe_Classes joined with Recipes) and then work outward. Because some rows in Recipe_Classes might not have any matching rows in Recipes, this first JOIN returns rows that have a Null value in RecipeClassID. Looking back at Figure 9–12, you can see that there's a one-to-many relationship between Recipe_Classes and Recipes. Unless some recipes exist that haven't been assigned a recipe class, we should get all the rows from the Recipes table anyway! The next JOIN with the Recipe_Ingredients table also asks for a LEFT OUTER JOIN. We want all the rows, regardless of any Null values, from the previous JOIN (of Recipe_Classes with Recipes) and any matching rows in Recipe_Ingredients. Again, because some rows in Recipe_Classes might not have matching rows in Recipes or some rows in Recipes might not have matching rows in Recipe_Ingredients, several of the rows might have a Null in the IngredientID column from the Recipe_Ingredients table. What we're doing with both JOINs is "walking down" the one-to-many relationships from Recipe_Classes to Recipes and then from Recipes to Recipe_Ingredients. So far, so good. (By the way, the final INNER JOIN with Measurements is inconsequential -- we know that all Ingredients have a valid MeasureAmountID.) Now the trouble starts. The final RIGHT OUTER JOIN asks for all the rows from Ingredients and any matching rows from the result of the previous JOINs. Remember from Chapter 5 that a Null is a very special value -- it cannot be equal to any other value, not even another Null. When we ask for all the rows in Ingredients, the IngredientID in all these rows has a non-Null value. None of the rows from the previous JOIN that have a Null in IngredientID will match at all, so the final JOIN throws them away! You will see any ingredient that isn't used yet in any recipe, but you won't see recipe classes that have no recipes or recipes that have no ingredients. If your database system decides to solve the query by performing the JOINs in a different order, you might see recipe classes that have no recipes and recipes that have no ingredients, but you won't see ingredients not yet used in any recipe because of the Null matching problem. Some database systems might recognize this logic problem and refuse to solve your query at all -- you'll see something like an "ambiguous OUTER JOINs" error message. The problem we're now experiencing results from trying to "walk back up" a many-to-one relationship with an OUTER JOIN going in the other direction. Walking down the hill is easy, but walking back up the other side requires special tools. What's the solution to this problem? Read on to the next section to find out! <div class="imagecaption alignLeft"> <img src="https://cdn.ttgtmedia.com/rms/onlineimages/sql_server-sql_queries_for_mere_mortals.png" alt=""> </div> This chapter excerpt from SQL Queries for Mere Mortals: A Hands-On Guide to Data Manipulation in SQL (2nd Edition) by John L. Viescas and Michael J. Hernandez, is printed with permission from <a href="https://www.informit.com/imprint/index.aspx?st=61085" target="_blank" rel="noopener">Addison-Wesley Professional</a>, Copyright 2007. Click here for the <a href="https://cdn.ttgtmedia.com/searchSQLServer/downloads/Outer_Joins_Query_book.pdf" target="_blank" rel="noopener">chapter download</a> or <a href="https://www.amazon.com/SQL-Queries-Mere-Mortals-Hands/dp/0321444434" target="_blank" rel="noopener">purchase the book here</a>. In this book excerpt, you'll learn LEFT OUTER JOIN vs. RIGHT OUTER JOIN techniques and find various examples for creating SQL queries that incorporate OUTER JOINs. https://cdn.ttgtmedia.com/visuals/searchMobileComputing/wireless_networking/mobilecomputing_article_022.jpg https://www.techtarget.com/searchdatamanagement/feature/How-to-use-the-LEFT-vs-RIGHT-OUTER-JOIN-in-SQL Tue, 02 Jul 2019 00:00:00 GMT Using a LEFT OUTER JOIN vs. RIGHT OUTER JOIN in SQL Regardless of the size of your organization, there's always a wide variety of items to keep track of in database systems like SQL Server -- user logins, data access, security protections and more. Collecting and maintaining information for database audits can be a daunting task, but SQL Server provides tools to help make it manageable, according to IT architect and author K. Brian Kelley. In a recent webinar, Kelley discussed the auditing features that come with the database software and how database administrators (DBAs) can utilize them to create a SQL Server audit trail that documents the operational minutiae in systems for <a href="https://searchsqlserver.techtarget.com/feature/5-trends-for-SQL-Server-environments-as-SQL-Server-2019-looms">security, regulatory compliance and administration</a> uses. One of the many auditing tools that SQL Server offers is the default trace. That feature is in maintenance mode, and Microsoft doesn't recommend using it in new applications. But it can still provide some useful information, according to Kelley, a former SQL Server DBA who currently works as a data architect at AgFirst Farm Credit Bank in Columbia, S.C. For example, the default trace mechanism can be used to audit user logins to check for application problems or potential security issues. "Out of the box, SQL Server makes it easy to catch failed logins," thanks largely to the default trace, Kelley said. The default trace's main job is to capture all changes to SQL Server's underlying configuration and schemas on a temporary basis. Kelley noted that the feature's name is a bit of a misnomer, as it actually consists of four or five small trace files. SQL Server rolls over these files as needed when their storage capacity fills up. As a result, the default trace is essentially a rolling set of audit records, he said. <h3>Extended Events offers a new audit track</h3> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> Out of the box, SQL Server makes it easy to catch failed logins. </figure> <figcaption> K. Brian Kelley Data architect, AgFirst Farm Credit Bank </figcaption> </div> </blockquote> For auditing in new applications, Microsoft recommends using its SQL Server Extended Events performance monitoring tool instead of the default trace feature. Extended Events was introduced in SQL Server 2008 to reduce the processing impact of <a href="https://searchsqlserver.techtarget.com/tip/SQL-Server-performance-tuning-best-practices-for-DBAs">database performance tuning</a> and auditing processes. The technology became more useful after a set of GUIs was added to it in SQL Server 2012. Similarly, Kelley stressed the importance of using Extended Events to create customized traces for a SQL Server audit trail instead of running SQL Server Profiler or server-side traces, two other older auditing mechanisms. Users can still build client-level traces with SQL Server Profiler and server-side traces with SQL Trace, but Microsoft has <a href="https://www.techtarget.com/whatis/definition/deprecated">deprecated</a> those tools. Traces built with them tend to be fairly heavy on the performance side, Kelley said, and they were replaced by Extended Events in SQL Server 2012. "If you want to audit things that are new features in any of the versions after SQL Server 2008 R2, the only way you can audit for it, the only way you can detect it and use it, is with Extended Events," he noted. <h3>Options for recording database logins</h3> In the case of logins, SQL Server is automatically configured to record all failed ones, but users can customize it to fit their needs, Kelley said. The system can be reconfigured to track all successful logins or both failed and successful attempts. If you have an alternative tracking system, you can even disable SQL Server's mechanism. Kelley warned, though, that caching successful logins on a busy system would be difficult because the event log would quickly start overwriting the data. Changing SQL Server's settings for auditing logins is relatively easy, Kelley said. First, go into <a href="https://searchsqlserver.techtarget.com/definition/Microsoft-SQL-Server-Management-Studio-SSMS">SQL Server Management Studio</a> and right-click on Server. Choose Properties, then select Security. The Security tab allows you to choose between four login auditing options -- None, Failed logins only, Successful logins only or Both failed and successful logins. SQL Server needs to be restarted after any changes are made; otherwise, it won't collect the data the way you want it to, he said. <h3>Alternative auditing tools</h3> Additional auditing mechanisms available to DBAs include Policy-Based Management, which Kelley described as "basically group policy for SQL Server." The Policy-Based Management feature can be used to enforce policy settings in SQL Server's database engine, but he said it's best utilized as a "check system," as it can verify almost anything in a SQL Server instance. Kelley considers Policy-Based Management to be under-utilized by most organizations. There are limits to what the tool can do, though. He cautioned that it's primarily a "detective control," as it only alerts DBAs to problems after they've already occurred. Additionally, Policy-Based Management isn't able to capture who did what in a SQL Server system; it's intended for ensuring that the configuration stays the way you want it to and alerting you when something is changed. <a href="https://www.techtarget.com/whatis/definition/Data-Definition-Language-DDL">Data definition language</a> triggers and logon triggers can also be used to help build a SQL Server audit trail in database systems. For example, they can be set to fire if someone attempts to log in to or alter a system. But Kelley said <a href="https://searchsqlserver.techtarget.com/tip/Pros-and-cons-of-using-SQL-Server-audit-triggers-for-DBAs">triggers should be used with caution</a>, as they can add processing overhead and unexpectedly block processes. "It's often better to just audit for the event happening," and then fix the issue afterwards, he added <a href="https://www.mssqltips.com/sql-server-video/774/introduction-to-auditing-features-in-sql-server/" target="_blank" rel="noopener">in the webinar</a>, which was posted on the MSSQLTips website. Change data capture (CDC) wasn't originally intended for auditing, according to Kelley. Its original intent was to record data changes in SQL Server tables so users only had to process what had been changed to update a data warehouse. Nonetheless, Kelley said CDC can be used when auditing, as it also allows DBAs to read database log files. However, he added that doing so isn't a supported use of the technology and should only be done as a last resort. One of the <a href="https://searchsqlserver.techtarget.com/tip/SQL-Server-auditing-best-practices-3-key-questions-for-DBAs">key things database auditors want to know</a> is which users have access to a database and what kind of access those users have. Kelley said there are two main levels of security catalog views for principal users: sys.server_principals and sys.database_principals. The sys.server_principals one, in particular, corresponds to logins. He described the two catalog views as a DBA's "bread and butter" when trying to determine who actually has access to data sets and -- more important -- who is supposed to have access. SQL Server offers a set of built-in auditing tools that can help make the process of tracking logins and other database activities easier for database administrators. https://cdn.ttgtmedia.com/visuals/searchNetworking/virtualization/networking_article_018.jpg https://www.techtarget.com/searchdatamanagement/tip/Key-features-to-create-a-SQL-Server-audit-trail-in-databases Fri, 31 May 2019 11:48:00 GMT Key features to create a SQL Server audit trail in databases Although there was significant fanfare surrounding SQL Server Query Store's initial release in SQL Server 2016, many companies have been hesitant about adopting the performance monitoring and troubleshooting tool, according to consultant Andy Warren. Ironically, that's largely because of its own potential impact on the performance of SQL Server systems. When Query Store is activated, it captures runtime statistics and other information about all submitted queries and query execution plans on a per-database basis. This allows users to turn the feature on for all of the databases in a database server or only specific ones, based on workload priorities or a desire to minimize the overall Query Store performance impact in their <a href="https://searchsqlserver.techtarget.com/feature/5-trends-for-SQL-Server-environments-as-SQL-Server-2019-looms">SQL Server environments</a>. In a recent webinar, Warren described Query Store as a set of tables that contain detailed information about the queries run against a database. Once it's turned on, the tool runs 24/7 and saves the information in a way that's easy for users to access and understand, said Warren, president of consultancy Fourdeuce Inc. in Orlando, Fla. Query Store can be used to detect regressions -- cases in which <a href="https://searchsqlserver.techtarget.com/tip/SQL-Server-performance-tuning-best-practices-for-DBAs">query performance slows down</a>. It then allows users to choose their own query plans instead of relying on SQL Server's Query Optimizer to do so. For example, they can force a previous plan saved in Query Store to execute in place of the current one to reverse a regression. The tool can also help research database usage patterns, though Warren said users need to have the SQL Server Profiler or Extended Events monitoring software installed in order to do that. As helpful as Query Store sounds, Warren said that it's underutilized. Many SQL Server database administrators (DBAs) already use third-party products that perform most of Query Store's tasks in a more user-friendly manner, he said <a href="https://www.mssqltips.com/sql-server-video/764/introduction-to-sql-server-query-store/" target="_blank" rel="noopener">in the webinar</a>, which was posted on the MSSQLTips website and sponsored by IT management tools vendor Quest Software Inc. <h3>Query Store performance impact a user concern</h3> Warren noted that potential users are also intimidated by the perceived performance impact of Query Store. Because the tool runs all the time if it's on, it can take system resources away from other programs. According to Warren, DBAs are likely to assume that running Query Store will increase processing overhead in their database servers -- and often with good reason. For those DBAs who are curious about using Query Store but want to assess its performance impact first, Warren recommended experimenting with the tool during off-hours. "This is something you turn on at six o'clock on Thursday night," he said. "Turn it on after the peak time of day." <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-query_store_best_practices.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-query_store_best_practices_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-query_store_best_practices_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-query_store_best_practices.png 1280w" alt="SQL Server Query Store best practices from Microsoft" height="380" width="560"> <div class="main-article-image-enlarge"> </div> </figure> Running Query Store does take up resources when it gathers information on queries and builds out its data store inside a <a href="https://searchsqlserver.techtarget.com/tip/How-to-resolve-and-avoid-deadlocks-in-SQL-Server-databases">SQL Server database</a>, and Warren said the tool may well use more CPU cycles than originally anticipated. That's perfectly normal, despite the possible performance hit it can cause, he assured webinar listeners. Warren cautioned, though, that DBAs typically will be guessing upfront at how much disk space is needed for the data collected by Query Store, which has a default capacity of 100 MB. The worst-case scenario, he said, is that the tool silently switches to <a href="https://www.techtarget.com/searchwindowsserver/definition/Read-only">read-only</a> mode after reaching its storage limit; if so, it stops collecting data, which affects the accuracy of query performance analysis. Because of this, Warren recommended checking the disk space once a week for the first month. <h3>How to turn on Query Store in SQL Server</h3> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> Turn it on, let it run for a little while, turn it off. That's a great 'get comfortable' exercise for learning about Query Store. </figure> <figcaption> Andy WarrenPresident, Fourdeuce Inc. </figcaption> </div> </blockquote> To enable Query Store to run in a database, right-click on that database in SQL Server Management Studio's Object Explorer UI, then click on Properties and open the Query Store page from the Database Properties dialog window. You then have the option to change the Operation Mode setting from Off to either Read Only or Read Write. Warren recommended choosing the latter setting, as the Read Only one should only be used if you've run out of room for new data or decide that you don't want to collect more. You can also change the intervals at which Query Store collects statistics and its allotted storage capacity in Object Explorer. Alternatively, all of these steps can be done via <a href="https://searchsqlserver.techtarget.com/definition/T-SQL">Transact-SQL</a> scripts using the ALTER DATABASE statement. Either way, Query Store is easy to activate and deactivate, Warren said. "Turn it on, let it run for a little while, turn it off," he advised. "That's a great 'get comfortable' exercise for learning about Query Store." Warren acknowledged that not every organization will have a use for the tool, partly because of the Query Store performance impact that users may see in their database servers. Nonetheless, he said it's still a technology that's worth knowing about for DBAs, especially because Query Store comes as a standard feature with SQL Server 2016 and beyond. Many IT teams hesitate to use SQL Server Query Store due to performance concerns. Consultant Andy Warren offers tips on how to test and get started with Query Store. https://cdn.ttgtmedia.com/visuals/IoTAgenda/business_of_iot/iotagenda_article_004.jpg https://www.techtarget.com/searchdatamanagement/tip/Check-SQL-Server-Query-Store-performance-impact-before-using Mon, 20 May 2019 16:10:00 GMT Check SQL Server Query Store performance impact before using SEATTLE -- Microsoft's latest string of Azure cloud database technologies looks to address a multitude of developers' needs that range from high-end workloads to edge applications. At the Microsoft Build 2019 Developer Conference here this week, Microsoft disclosed that Azure SQL Hyperscale is now generally available. The database raises the 4 TB limit on Azure SQL database sizes up to 100 TB. previewed in October, the Hyperscale tier will assist shops that previously had to split large Azure SQL workloads across multiple databases. Admins can now manage actions such as database recovery within a single large instance. Microsoft also a Hyperscale option for PostgreSQL into preview. It is based on the January <a href="https://www.techtarget.com/searchcloudcomputing/news/252456490/Microsoft-grabs-Citus-Data-expands-Azure-open-source-options">acquisition of Citus Data</a> and added as an extension to PostgreSQL, not a fork of the core codebase, which means that customers can keep in step with updates. The PostgreSQL expansion is Microsoft's latest push into <a href="https://www.techtarget.com/searchdatamanagement/news/252472641/Open-source-database-use-is-a-growing-trend">open source database</a> options for Azure. "We are doing great things with proprietary databases, but Azure is a platform for every developer," said Rohan Kumar, corporate vice president of Azure data platform, at Build. <a href="https://www.techtarget.com/searchcloudcomputing/news/252463609/Microsoft-hands-developer-advocates-keys-to-the-kingdom">Azure SQL Database serverless</a> is also in preview. The fully managed service is best for intermittent workloads on a single database, for scenarios such as development and test, gaming and e-commerce, Kumar said. SQL Database serverless is billed on a per-second basis, which saves customers money over less granular models, Kumar added. Chemical giant BASF uses Cosmos DB for a project that draws data from 20 million sensors spread across hundreds of sites around the world to compile data on metrics such as temperature, pressure and flow rates, said Robert Pack, a research scientist who joined Kumar onstage. "We're not looking to optimize production within the scope of a single plant, but rather optimize production across multiple sites and value chains," Pack said. "Exploring this very high-dimensional space and arriving at actual insights for our experts is a tremendous challenge." BASF has used Cosmos DB to build a graph representation of its footprint that provides thermo-physical and systems-level contexts for all the signals it collects. This provides a basis for analytics it runs with the help of Azure Data Explorer, according to Pack. "We're now able to analyze decades of high- time-series data from multiple instances of a process, distributed across the globe, to build models that not only are valid for one specific plant, but a technology as a whole," he said. Microsoft also <a href="https://www.techtarget.com/searchenterpriseai/news/252462719/Microsoft-AI-services-roadmap-to-be-delivered-at-Build">disclosed a preview</a> of Azure SQL Database Edge, an optimized version of the database geared to run on edge devices for IoT scenarios. <section class="section main-article-chapter" data-menu-title="Choosing the right database begins with the right discussions"> <h2 class="section-title">Choosing the right database begins with the right discussions</h2> The cohort of Azure cloud database updates at Microsoft Build are on the whole interesting but should be taken in the right context, said Jarrad Edwards, CTO of Roam Creative, a digital services consultancy with offices in Sydney, Berlin and Auckland, New Zealand. <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> I'm not a believer of just taking any database that looks like it will fit and pushing it into a project. It's got to be the right database that has the right benefits for the end use case. </figure> <figcaption> Jarrad EdwardsCTO, Roam Creative </figcaption> </div> </blockquote> "I'm not a believer of just taking any database that looks like it will fit and pushing it into a project," Edwards said in an interview after Kumar's session. "It's got to be the right database that has the right benefits for the end use case." Customers shouldn't be dazzled by new capabilities and neglect to have a solid, broad-based discussion with various constituents on the project's ultimate goal, he said. "Technology is only one side of it," Edwards said. "Myself, being a CTO, I would sit there and debate with a CIO, to talk about what their needs are so we can make sure we get a database that benefits everybody in the picture." Edwards recalled an application project he worked on several years ago. "We took a database that was rather read-heavy -- you could pull a lot of information out quickly in succession -- but as our application grew, we found it was actually very write-heavy," he said. "If we had spent more time upfront and actually discussed it as a whole ... we would have gotten to a solution so much quicker." </section> <section class="section main-article-chapter" data-menu-title="Azure cloud databases' all-things-to-everyone approach not unique"> <h2 class="section-title">Azure cloud databases' all-things-to-everyone approach not unique</h2> Of course, Microsoft isn't the only vendor that pushes a phalanx of <a href="https://www.techtarget.com/searchdatamanagement/definition/database-management-system">database options</a>, particularly open source ones. At the same time, cloud providers also steer customers to their own proprietary database services, said Doug Henschen, an analyst at Constellation Research. Microsoft, for example, touts Cosmos as being "<a href="https://docs.microsoft.com/en-us/azure/cosmos-db/mongodb-introduction" target="_blank" rel="noopener">wire-protocol compatible</a>" with open source options Cassandra and MongoDB, while AWS positions Aurora as a compatible replacement for MySQL and PostgreSQL. Compatibility in that sense has significant limits, according to Henschen. "Customers should keep in mind that proprietary, single-cloud services are not portable to other clouds or to conventional on-premises deployment," he said. It's also not a simple matter to migrate a deployment back to an open source database, if and when you want to move it back on premises or to another cloud. "The point is avoiding placing all bets on one cloud and being very choosy about using single-cloud services that might diminish deployment flexibility," Henschen said. </section> A raft of expansions across Microsoft's Azure cloud database tier adds capabilities that target high-end workloads and shops that prefer open source options. https://cdn.ttgtmedia.com/visuals/searchNetworking/virtualization/networking_article_019.jpg https://www.techtarget.com/searchcloudcomputing/news/252462956/Microsoft-expands-Azure-cloud-database-options Tue, 07 May 2019 17:49:00 GMT Microsoft expands Azure cloud database options SQL Server performance tuning is a complex topic due to how layered it is. While performance problems can often be traced back to data storage issues, the performance tuning process is a "holistic effort" that should be examined from all sides of a SQL Server environment, according to database consultant Joey D'Antoni. For starters, query tuning is not the same thing as performance tuning overall, D'Antoni said in a recent webinar on SQL Server performance tuning best practices. It's important that queries run quickly and efficiently, but equating the two can lead to a misdiagnosis of performance issues in a <a href="https://searchsqlserver.techtarget.com/feature/5-trends-for-SQL-Server-environments-as-SQL-Server-2019-looms">SQL Server system</a>, he cautioned. Database administrators (DBAs) still need to understand the impact that queries have on overall <a href="https://www.techtarget.com/searchdatamanagement/feature/Three-indicators-that-could-signal-database-performance-issues">database server performance</a>, though -- and vice versa. "You can have all the query optimization in the world, but if you're using junk hardware, it doesn't matter," said D'Antoni, who is a principal consultant at Denny Cherry & Associates Consulting in Oceanside, Calif. On the other hand, sometimes issues that look like hardware problems are instead directly related to badly written queries, he added. <h3>Establish a performance baseline</h3> A performance baseline is the most important thing to measure for <a href="https://www.techtarget.com/searchitoperations/opinion/Database-performance-tuning-Five-ways-for-IT-to-save-the-day">properly tuning a database</a>, D'Antoni advised. And because having a baseline is integral to SQL Server performance tuning, DBAs should establish one early on, he said. Once you've done so, you can determine problem areas by comparing the baseline statistics to SQL Server's current performance. CPU utilization isn't necessarily the key aspect of a performance baseline, but it is the largest factor and must be included, according to D'Antoni. Determining how the CPUs in a system are being utilized will show how busy your server is at any given time and, when checked against the baseline level, indicate which areas should be altered or fixed. For example, he said high CPU utilization is frequently caused by bad code or bad queries. Other metrics that should go into the performance baseline include I/O latency and how long it takes to run important reports or administrative commands like checkdb, D'Antoni said. While it's possible to create baselines yourself manually, he recommended using a <a href="https://www.techtarget.com/searchdatamanagement/feature/What-you-need-to-know-about-database-performance-software">performance monitoring tool</a> to speed the process. Cloud portals are also good sources of baseline information on database servers that run in the Azure, AWS or Google clouds. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-db_performance_tuning.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-db_performance_tuning_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-db_performance_tuning_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-db_performance_tuning.png 1280w" alt="To-do items for tuning database performance" height="463" width="560"> <figcaption> Database performance tuning to-do items </figcaption> <div class="main-article-image-enlarge"> </div> </figure> <h3>Know your SQL Server system resources</h3> D'Antoni said SQL Server performance tuning best practices also include knowing the limits of your system resources -- or, as he put it, "knowing what your computer is made of." In particular, system memory is finite and must be divided up among the different programs that need to run on a server. Data is pulled into RAM or the SQL Server buffer pool for processing, and SQL Server also uses memory to join queries together, rebuild indexes and sort data. <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> It's important to understand what waits matter and what waits are benign. </figure> <figcaption> Joey D'Antoniprincipal consultant, Denny Cherry & Associates Consulting </figcaption> </div> </blockquote> To avoid performance problems, be sure not to overallocate the available memory in a SQL Server system, D'Antoni said. He warned that if there isn't enough memory for workloads, you'll wind up putting stress on the storage devices in the system, potentially slowing down I/O throughput. Your network is another limited resource to take into account when tuning SQL Server performance, he added. In addition to checking things like CPU utilization, I/O latency and memory usage, D'Antoni advised SQL Server DBAs to use the wait statistics that database systems collect. SQL Server tracks and logs what each thread is waiting on whenever queries and other jobs are temporarily blocked from running. Aggregating these statistics can help you analyze and prevent performance bottlenecks. "It's important to understand what waits matter and what waits are benign," D'Antoni said. <h3>Performance tuning questions to ask</h3> In the <a href="https://www.mssqltips.com/sql-server-video/760/understanding-your-sql-servers-performance/" target="_blank" rel="noopener">webinar</a><a href="https://www.mssqltips.com/sql-server-video/760/understanding-your-sql-servers-performance/"></a>, which was hosted by database tools vendor Idera, D'Antoni said you also must ask -- and answer -- the following questions as part of the performance tuning process: <ul class="default-list"> <li>Is anything else running on my server?</li> <li>Is an antivirus program unintentionally hindering the progress of processing jobs?</li> <li>Is a <a href="https://searchservervirtualization.techtarget.com/definition/virtual-machine">virtual machine</a> (VM) running on the server?</li> </ul> Microsoft ships SQL Server and all related products in the same installation file. As a result, users can easily install technologies like SQL Server Analysis Services, Integration Services and Reporting Services on the same server as their SQL Server database engine. This isn't inherently a bad thing, in D'Antoni's view, but it takes memory and other system resources away from SQL Server itself. "In an ideal world, we'd run each component on its own individual server," he said. Unfortunately, licensing issues make that difficult, if not impossible. DBAs need to plan accordingly when working to tune SQL Server's performance, he added. While antivirus software is designed to help organizations avoid data breaches and other security problems, it can also slow down SQL Server performance, D'Antoni said. He advised that you make sure an antivirus program isn't scanning MDF, LDF or NDF files in a SQL Server database in order to reduce the amount of I/O operations being processed on the server. If you run SQL Server in VMware or another virtualized environment, your VM host and <a href="https://searchservervirtualization.techtarget.com/definition/hypervisor">hypervisor</a> configuration can negatively affect system performance, too, D'Antoni cautioned. In the case of VMware, DBAs should have access to <a href="https://www.techtarget.com/searchvmware/definition/VMware-VirtualCenter">vCenter Server</a>, the virtualization platform's server management software, so they can see how the hypervisor is set up and what resources are allocated to it. "If you're running VMware, you want to have insight into vCenter, even if it's read-only," he said. Tuning database performance is a complex process, but consultant Joey D'Antoni details a list of SQL Server performance tuning best practices that can make it easier. https://cdn.ttgtmedia.com/visuals/searchDataManagement/integration_technology/datamanagement_article_019.jpg https://www.techtarget.com/searchdatamanagement/tip/SQL-Server-performance-tuning-best-practices-for-DBAs Mon, 22 Apr 2019 13:43:00 GMT SQL Server performance tuning best practices for DBAs SAN FRANCISCO -- Google's decision to offer Microsoft SQL Server as a managed service is designed to attract customers with certain Windows workloads, and adds to Google Cloud Platform's overall appeal. SQL Server users could already run the database themselves on Google Cloud Platform (GCP) via VMs, but Google will fully manage the upcoming service through its Cloud SQL offering, which already features PostgreSQL and MySQL. Google's managed SQL Server service will support all editions of SQL Server 2017, which also <a href="https://www.techtarget.com/searchsoftwarequality/tip/Common-software-compatibility-issues-and-how-to-fix-them">has backward compatibility</a> with older versions of the database, said Dominic Preuss, director of product management for Google Cloud, at the Cloud Next conference here this week. AWS has offered a similar service through its Relational Database Service for years. Moreover, Microsoft has worked since 2009 on its Azure SQL managed service. Microsoft's effort has endured some fits and starts over the years. Customers that wanted to <a href="https://www.techtarget.com/searchcloudcomputing/news/252454183/Azure-database-updates-court-on-premises-installed-base">move very large SQL Server databases</a> to the cloud had to run them on Azure's VM-based service or break them apart into multiple pieces, given Azure SQL's size limitations. That changed drastically in October, when Microsoft raised the Azure SQL database limit to 100 TB, up from 4 TB. The company also introduced Managed Instance, which is similar in function to on-premises SQL Server. <div class="imagecaption alignRight"> <img src="https://cdn.ttgtmedia.com/rms/onlineImages/henschen_doug.jpg " alt="Doug Henschen, analyst, Constellation Research">Doug Henschen </div> Currently, <a href="https://cloud.google.com/sql/" target="_blank" rel="noopener">Google Cloud SQL supports</a> MySQL and PostgreSQL instances with up to 10 TB of storage capacity, which is well below Azure SQL's limits. Google has yet to determine scalability limits for its managed SQL Server, but most databases are less than a terabyte, Preuss said. Google's intent is not to support the largest and most complex SQL Server workloads, according to Preuss. "Really complicated, advanced things with SQL Server?" he said. "That's something you're going to want to do on your own." Google's managed SQL Server service has started its initial test phase, with general availability planned later this year, Preuss said. <section class="section main-article-chapter" data-menu-title="Google's SQL Server service reflects multi-cloud applicability"> <h2 class="section-title">Google's SQL Server service reflects multi-cloud applicability</h2> Google's managed SQL Server service may not compete directly with Microsoft, however. With Azure SQL, Microsoft also supports a series of hybrid-deployment needs, such as backup and recovery and burst capacity from on-premises Microsoft SQL Server deployments into the cloud, said Doug Henschen, an analyst at Constellation Research based in Cupertino, Calif. <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> Google isn't going to support these hybrid situations. The focus is on supporting workloads that run on Microsoft SQL Server in GCP. </figure> <figcaption> Doug HenschenAnalyst, Constellation Research </figcaption> </div> </blockquote> "Google isn't going to support these hybrid situations," he said. "The focus is on supporting workloads that run on Microsoft SQL Server in GCP." So why would SQL Server customers rent space on Google Cloud instead of Azure? Some companies have <a href="https://www.techtarget.com/searchcloudcomputing/news/252457171/UK-news-org-plots-move-from-AWS-to-Google-Cloud-Platform">picked GCP as their cloud</a>, yet there are many preferred apps that run on Microsoft SQL Server, such as marketing applications, Henschen said. Also, many organizations have lots of marketing data on GCP, including Google Analytics, advertising and search-related data, so it makes some sense to keep the marketing data and the database on the same cloud, Henschen said. </section> Google's decision to add SQL Server to its Cloud SQL family of managed database services is an appeal to Google Cloud Platform customers with ample Windows workloads. https://cdn.ttgtmedia.com/visuals/searchCloudComputing/public_cloud/cloudcomputing_article_026.jpg https://www.techtarget.com/searchcloudcomputing/news/252461560/Google-expands-Windows-support-with-managed-SQL-Server Thu, 11 Apr 2019 17:03:00 GMT Google expands Windows support with managed SQL Server The first step in learning how to avoid deadlocks in SQL Server databases is preparedness. Knowing about deadlocks -- in particular, what causes them -- is the key to preventing them and resolving those that do occur, according to SQL Server experts. Deadlocking isn't the same thing as blocking. The latter happens normally during database operations when a transaction tries to access a system resource that has been locked by another transaction. The <a href="https://searchsqlserver.techtarget.com/feature/5-trends-for-SQL-Server-environments-as-SQL-Server-2019-looms">SQL Server system</a> blocks the second transaction until the first completes its process and releases the lock -- a situation that can be resolved by SQL Server itself without requiring any manual intervention. SQL Server <a href="https://www.techtarget.com/whatis/definition/deadlock">deadlocks</a>, on the other hand, happen when two transactions block each other from resources they need to use, resulting in what's referred to as a deadly embrace that stops both transactions from continuing. SQL Server can automatically clear deadlocks, but only by terminating one of the transaction threads, and the intervention process to ensure that deadlocks don't recur can be lengthy and complex for database administrators (DBAs). <section class="section main-article-chapter" data-menu-title="What causes deadlocks in databases?"> <h2 class="section-title">What causes deadlocks in databases?</h2> Thomas LaRock, "head geek" at management tools vendor SolarWinds in Austin, Texas, wrote in a June 2016 blog post that there are four primary -- and interrelated -- causes of deadlocks in SQL Server: application code, the <a href="https://www.techtarget.com/searchdatamanagement/tip/Relational-database-design-tips-to-boost-performance">database schema design</a>, the resulting access patterns and the <a href="https://www.theserverside.com/blog/Coffee-Talk-Java-News-Stories-and-Opinions/How-transaction-isolation-levels-work">transaction isolation level settings in a database</a>. Deadlocks don't only occur in databases that contain large tables with indexes, LaRock added -- they can also happen in small tables. He said databases that encounter deadlocks can be seen as performance bottlenecks by end users, which puts pressure on SQL Server DBAs and application developers to work together "to track down the root cause and fix the issue." <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineImages/SQL_server_deadlock_blockers.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineImages/SQL_server_deadlock_blockers_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineImages/SQL_server_deadlock_blockers_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineImages/SQL_server_deadlock_blockers.png 1280w" alt="Microsoft's tips on minimizing SQL Server deadlocks" data-credit="TechTarget" height="354" width="558"> <figcaption> Tips from Microsoft on how to avoid SQL Server deadlocks </figcaption> <div class="main-article-image-enlarge"> </div> </figure> </section> <section class="section main-article-chapter" data-menu-title="Ways to avoid SQL Server deadlocks"> <h2 class="section-title">Ways to avoid SQL Server deadlocks</h2> Avoiding deadlocks in SQL Server is easier said than done, but it's not an impossible task. In a December 2017 blog post, Victor Simon, a data recovery expert at DataNumen in Hong Kong, pointed to poor database design as a common cause of deadlocks. To help minimize the deadlock risk, he said a SQL Server DBA should create a well-defined order for how concurrent transactions access database objects, along with a clear set of rules to govern the process. DBAs should also restrict users from inputting data while a transaction is being processed, Simon said. In addition, deadlocks are sometimes created because tables are locked when users run queries against them if a database is configured with the READ COMMITTED transaction isolation level, SQL Server's default setting. Additional queries have to wait for the lock to be released, but Simon said using the NOLOCK table hint in <a href="https://searchsqlserver.techtarget.com/definition/T-SQL">T-SQL</a> statements allows DBAs to override table locking. Using bound connections is another way to help avoid deadlocks in SQL Server, Simon wrote. That enables an application to open multiple cooperative connections to a database that can work with each other on locking so they don't block each other, reducing the likelihood of problems like deadlocking and database corruption. </section> <section class="section main-article-chapter" data-menu-title="Deadlock removal with SQL Server's lock monitor"> <h2 class="section-title">Deadlock removal with SQL Server's lock monitor</h2> SQL Server comes with a lock monitor feature that can detect and end deadlocks. It periodically searches for them in a database instance. The default interval between searches is every five seconds, but Microsoft says that drops to as low as 100 milliseconds if any deadlocks are found, with the new interval varying based on the frequency of the deadlocks. <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> DBAs must still investigate deadlocks to determine their cause so steps can be taken to avoid them in the future. </figure> </div> </blockquote> When the lock monitor detects multiple application threads trapped in a deadlock, it selects one of them to be the deadlock victim. That thread's processing jobs are terminated, and the ongoing transaction is rolled back in the database. Doing so releases the locks held by the deadlock victim, which allows the other blocked session -- or sessions -- to resume processing. The lock monitor, by default, determines the deadlock victim based on the rollback cost, choosing the thread that will be least expensive to roll back from a processing standpoint. DBAs can also use the SET DEADLOCK_PRIORITY statement to assign priority levels to threads in the event of deadlocks. The session with the lowest priority will then be chosen as the deadlock victim. If multiple sessions have the same deadlock priority, the lock monitor will revert to choosing the one with the lowest rollback cost. </section> <section class="section main-article-chapter" data-menu-title="Info about deadlocks to help avoid them"> <h2 class="section-title">Info about deadlocks to help avoid them</h2> Though it may seem more convenient to rely solely on the lock monitor to do its job, DBAs must still investigate deadlocks to determine their cause so steps can be taken to avoid them in the future. There are three primary methods of getting information about deadlocks -- known as the deadlock graph -- from SQL Server. Using trace flag 1222 in T-SQL returns information captured in SQL Server's error log after deadlocks occur; trace flag 1204 can also be used, but 1222 provides more details. The trace flags were once the only real way to access the deadlock graph info, according to Grant Fritchey, a product evangelist at Redgate, a SQL Server tools vendor based in Cambridge, England. To capture information on deadlocks as they occur, however, trace flag 1222 must be permanently enabled, Fritchey wrote in an article on Redgate's website. With the advent of newer, more efficient methods, he recommended only using the trace flag as a last resort to aid in deadlock prevention. The other approaches include using the system_health session in SQL Server's built-in Extended Events performance monitoring tool to view deadlock information. Extended Events was introduced <a href="https://searchsqlserver.techtarget.com/tip/4-strategies-to-beat-the-SQL-Server-2008-support-cutoff">in SQL Server 2008</a>, and Microsoft added the deadlock graph to the system_health session in SQL Server 2012. According to <a href="https://docs.microsoft.com/en-us/sql/relational-databases/sql-server-transaction-locking-and-row-versioning-guide?view=sql-server-2017" target="_blank" rel="noopener">Microsoft's documentation</a>, the session is enabled by default and automatically captures information about the processes and system resources involved in deadlocks, plus a list of deadlock victims. DBAs can also use Microsoft's SQL Server Profiler tool to save deadlock graphs for viewing and analysis in <a href="https://searchsqlserver.techtarget.com/definition/Microsoft-SQL-Server-Management-Studio-SSMS">SQL Server Management Studio</a>, although the profiler is now in maintenance mode and not recommended for use with new applications. As an alternative, users can turn to third-party performance monitoring tools to help in their efforts to resolve and avoid deadlocks in SQL Server. </section> Deadlocks are a real hindrance to SQL Server users, but database administrators can avoid them by taking steps to limit them and stop them from recurring. https://cdn.ttgtmedia.com/visuals/searchOracle/sunoracle_systems/oracle_article_025.jpg https://www.techtarget.com/searchdatamanagement/tip/How-to-resolve-and-avoid-deadlocks-in-SQL-Server-databases Wed, 27 Mar 2019 14:49:00 GMT How to resolve and avoid deadlocks in SQL Server databases A compliance audit can sometimes feel like a database administrator's worst nightmare, but it doesn't have to be. The key to acing a SQL Server database audit is preparedness; planning ahead and carefully monitoring the use of your data from the get-go can take some of the pain out of the auditing process. That's according to Bullett Manale, vice president of sales engineering at Idera. In a webinar hosted by the database tools vendor, Manale discussed SQL Server auditing best practices -- in particular, what the auditors really want to know and how to best answer their questions. According to Manale, auditors typically have three main questions for database administrators (DBAs): <ol class="default-list"> <li>Who has access to the <a href="https://www.techtarget.com/searchdatamanagement/tip/Three-factors-for-protecting-sensitive-data-in-the-GDPR-era">sensitive data in a database</a>?</li> <li>Are the right people accessing the data?</li> <li>Is the audit trail that's being used to validate the access controls reliable?</li> </ol> To avoid problems during compliance-related database audits, SQL Server DBAs should be able to answer all of these questions, said Manale, who addressed them individually in more detail. <h3>Tracking access to sensitive SQL Server data</h3> At the outset of a <a href="https://searchcompliance.techtarget.com/definition/compliance-audit">compliance audit</a>, the auditors will want to know who has access to sensitive data and -- more importantly -- how they gained that access, Manale said. This really boils down to knowing who should and shouldn't have access to the data, he added. <figure class="main-article-image half-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/sql-database_audit.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/sql-database_audit_half_column_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/sql-database_audit_half_column_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/sql-database_audit.png 1280w" alt="Key info for SQL Server database audits" height="281" width="279"> <figcaption> Information that SQL Server DBAs need for database audits </figcaption> <div class="main-article-image-enlarge"> </div> </figure> Knowing exactly where relevant data resides in a database is the key to determining who has access to it, according to Manale. As part of SQL Server auditing best practices, DBAs need to understand how user permissions are applied to the parts of databases where sensitive data is stored, he said. Tools that can help are available -- Manale noted that they enable DBAs to isolate and identify users, as well as their rights and permissions, painting a complete picture of who has access to what in a database. Before moving onto their second question, auditors typically will need to know that all this information has been validated by an organization's compliance officer, Manale said. <h3>Data access for the right people only</h3> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> Even if the audit trail shows compliance, many auditors will not trust the information without some form of integrity check of the audit itself. </figure> <figcaption> Bullett Manalevice president of sales engineering, Idera </figcaption> </div> </blockquote> Auditors also want to make sure that workers are utilizing sensitive data properly. In order to do so, Manale said, they'll ask for proof that the data is only being accessed by the right people -- qualified individuals whose job requires the data in question. It's equally important to be able to prove that the data can't be accessed by the wrong people, he added. To assure auditors that the data access situation is under control, SQL Server auditing best practices include being able to provide detailed records and reports on when data was accessed and by whom. Manale said the main challenge with that for DBAs is not only keeping track of the who, what, when and where of every transaction related to particular data sets, but also maintaining those records and being able to pull them up as needed. <h3>A complete and accurate audit trail</h3> Finally, auditors will want to ensure that the audit trail being used to validate the data access controls hasn't been tampered with. In order to prove that, DBAs need to be able to show that the audit data hasn't changed, Manale said. This can be done using hash functions or cyclic redundancy checks, but Manale cautioned that it can be an especially complicated task because the <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-integrity-protection-spurs-greater-security-spending">integrity of the data</a> in a database is essential to passing a SQL Server audit. "Even if the audit trail shows compliance, many auditors will not trust the information without some form of integrity check of the audit itself," he said. To adhere to SQL Server auditing best practices, Manale recommended that DBAs collect and maintain a full audit trail for transactions relating to the sensitive data in a database. Data should be kept in accordance with a standard <a href="https://www.techtarget.com/searchdatabackup/definition/data-retention-policy">data retention policy</a> -- typically, for a minimum of seven years, he said. He added that real-time controls should also be put in place to alert DBAs when transactions don't meet corporate data usage standards so they can put a stop to activity that could create compliance risks. Acing a SQL Server database audit starts with careful monitoring of how sensitive data is accessed and used so you can answer the top questions that auditors ask. https://cdn.ttgtmedia.com/visuals/searchCloudComputing/use_cases/cloudcomputing_article_019.jpg https://www.techtarget.com/searchdatamanagement/tip/SQL-Server-auditing-best-practices-3-key-questions-for-DBAs Thu, 21 Feb 2019 09:55:00 GMT SQL Server auditing best practices: 3 key questions for DBAs The <a href="https://searchsqlserver.techtarget.com/definition/SQL-Server">SQL Server</a> landscape is changing fast. Less than two years after the SQL Server 2017 release, Microsoft is poised to roll out SQL Server 2019. You can tell a lot about where it thinks the database market is headed based on the big data management and analytics features showcased in the new version, which is available now in preview mode. While <a href="https://www.techtarget.com/searchdatamanagement/definition/big-data">big data</a> deployments likely will take up a bigger part of the planning and operational processes in SQL Server environments, that's certainly not the only change in store over the course of this year. Let's take a look at some of the major trends that SQL Server database administrators (DBAs) and developers can expect in 2019. <h3>Big data is becoming SQL Server data</h3> First, to go back to where we started: Once just a blip on the SQL Server DBA's radar, big data and the advanced analytics it enables are becoming far more <a href="https://www.techtarget.com/searchdatamanagement/news/252448331/Big-data-tooling-rolls-with-the-changing-seas-of-analytics">common in business applications</a>. Businesses today often employ a combination of structured relational data and nonrelational data that's usually unstructured or semistructured; in many cases, they need to consolidate all of that information for analytics and reporting uses. Reflecting this change, enhanced big data integration is a primary focus of SQL Server 2019. It increases the number of external data sources that Microsoft's PolyBase <a href="https://www.techtarget.com/searchdatamanagement/definition/data-virtualization">data virtualization</a> software can access and allows users to build big data clusters that combine SQL Server with the Apache Spark processing engine and the Hadoop Distributed File System. These additions will help businesses incorporate modern data processing technologies and AI-enhanced applications into their SQL Server environments. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-2019_features.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-2019_features_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-2019_features_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-2019_features.png 1280w" alt="Capsule list of SQL Server 2019's key new features" height="245" width="560"> <figcaption> A summary of key new features in SQL Server 2019 </figcaption> <div class="main-article-image-enlarge"> </div> </figure> <h3>More SQL Server on Linux and in containers</h3> While its current adoption rate is still rather low compared to SQL Server on Windows, there's no doubt that SQL Server on Linux will continue to gain users in the coming year. With SQL Server 2019, Microsoft is bringing the <a href="https://searchsqlserver.techtarget.com/tip/SQL-Server-2019-improves-Linux-container-support">feature set available on Linux</a> into parity with what's in the Windows version, including new support for data replication, distributed transactions and the company's machine learning software. SQL Server on Linux could be a more cost-efficient option for many businesses than both its Windows counterpart and Oracle Database. In 2019, more businesses likely will also evaluate whether to put their SQL Server environments in containers. Docker containers enable fast and easy SQL Server deployments. Developers can fire up new <a href="https://searchsqlserver.techtarget.com/tip/2-ways-to-attach-SQL-Server-database-files-to-Linux-containers">containerized SQL Server instances</a> -- complete with databases -- in a matter of seconds. The ability to do so is becoming more important as organizations adopt fast-moving DevOps processes and continuous integration and continuous deployment methodologies. As the use of SQL Server in containers grows, the need to manage and orchestrate them becomes essential. <a href="https://www.techtarget.com/searchitoperations/definition/Google-Kubernetes">Kubernetes</a> has emerged as the container orchestration tool of choice for many organizations, and it's an important part of SQL Server 2019. The Linux version of the new release enables users to create Always On Availability Groups for SQL Server in Kubernetes-managed containers, which provides a more robust high-availability setup than the container instance failover support in SQL Server 2017. <h3>Moving SQL Server databases to the cloud</h3> While DBAs typically aren't eager to embrace the cloud, the migration of SQL Server environments to the cloud will certainly continue to grow in 2019 and beyond. The cloud offers cost and scalability benefits that are hard to keep ignoring, while barriers to cloud deployments continue to shrink -- in general and for SQL Server users in particular. Microsoft has adopted a cloud-first development model under which new SQL Server features are tested and implemented in its Azure SQL Database cloud service before they're moved into the on-premises product. Most <a href="https://searchsqlserver.techtarget.com/definition/T-SQL">Transact-SQL</a> features are now fully supported in both SQL Server and Azure SQL Database, and Microsoft said the Managed Instance version of the cloud service it launched in late 2018 can run existing SQL Server applications <a href="http://www.sparkhound.com/blog/azure-managed-sql-instances-a-distinct-cloud-based-offering" target="_blank" rel="noopener">without any changes</a>. <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> It's abundantly clear that the meaning of data in a business sense continues to expand, and the capabilities of SQL Server are continuing to grow in response. </figure> </div> </blockquote> Also, both SQL Server 2017 and 2019 have built-in hybrid cloud features that can span on-premises and cloud database instances. <h3>Data growth remains rampant for users</h3> Most businesses face unprecedented data growth. Different organizations have adopted diverse solutions to cope with that. Some simply continue to buy more storage, while others have adopted more efficient storage systems that support flash memory, data deduplication and compression or hybrid cloud setups in which infrequently accessed data and applications are moved to lower-cost cloud storage. Whatever course is taken, storage management should be a top to-do item for SQL Server DBAs, even more so if their organizations plan to use the big data features in SQL Server 2019. <h3>Time to update your SQL Server skills</h3> It's abundantly clear that the meaning of data in a business sense continues to expand, and the capabilities of SQL Server are continuing to grow in response. The new features call for a shift in how we think about SQL Server and the data it holds; it's also imperative to learn how to best use Microsoft's rapidly evolving technologies to ensure that SQL Server environments can meet changing business requirements. Learning how to integrate SQL Server with big data platforms and AI, machine learning and IoT tools will be a priority for many SQL Server DBAs. It also makes a lot of sense to bone up on Linux basics, Docker containers and Kubernetes, and to keep up with Azure SQL Database, SQL Server's hybrid cloud capabilities and related new cloud services offered by Microsoft. SQL Server is undergoing new changes, as Microsoft prepares to release the 2019 version of the database software. Other changes are also on tap for SQL Server users. https://cdn.ttgtmedia.com/visuals/searchSQLServer/microsoft_server_platforms/sqlserver_article_012.jpg https://www.techtarget.com/searchdatamanagement/feature/5-trends-for-SQL-Server-environments-as-SQL-Server-2019-looms Fri, 01 Feb 2019 09:02:00 GMT 5 trends for SQL Server environments as SQL Server 2019 looms Following closely on the heels of its SQL Server 2016 and 2017 releases, Microsoft is already preparing to release SQL Server 2019, and the company has added a surprising number of new capabilities in a short time. Enhanced big data integration is one of the major goals of the SQL Server 2019 release, which currently is available in preview mode. The new SQL Server big data clusters feature significantly broadens SQL Server's ability to query over data anywhere, providing better integration with big data systems over more supported platforms. But that's not all. The SQL Server 2019 relational database engine has many enhancements that enable it to provide better performance and improved security. Microsoft has also continued to enhance SQL Server on Linux, bringing it closer to feature parity with SQL Server on Windows. Let's dig into some of the most important new features in the upcoming SQL Server 2019 release. <h3>Big data clusters enable data virtualization</h3> SQL Server big data clusters enable deep integration with industry-standard big data sources, providing new <a href="https://www.techtarget.com/searchdatamanagement/definition/data-virtualization">data virtualization</a> capabilities. SQL Server 2019 big data clusters utilize the capabilities of PolyBase to enable data virtualization, which combines data from many sources without moving or copying it. Today's enterprise typically uses different types of data stored in a combination of relational and nonrelational data stores, and it usually needs to combine the data from both relational and big data sources. The highest-value enterprise data is stored in relational databases like SQL Server; that data often needs to be combined with big data sources, such as IoT data streams, which are primarily stored in Hadoop Distributed File System (<a href="https://www.techtarget.com/searchdatamanagement/definition/Hadoop-Distributed-File-System-HDFS">HDFS</a>). Organizations have typically used extract, transform and load (<a href="https://www.techtarget.com/searchdatamanagement/definition/Extract-Load-Transform-ELT">ETL</a>) processes to copy data into a single platform where reporting and analysis are performed. This process is batch-oriented, is slow and requires large amounts of storage. Using SQL Server's existing PolyBase technology, SQL Server 2019's big data clusters enable data virtualization that integrates data from multiple data sources without replicating or moving the data -- essentially combining high-value relational data with big data in a single query without the need to run any ETL process or use any extra storage. To enable this, the SQL Server 2019 database engine reads HDFS files natively. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-2019_features.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-2019_features_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-2019_features_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/onlineimages/sql_server-2019_features.png 1280w" alt="Three of the top features in SQL Server 2019" height="245" width="560"> <figcaption> Figure 1. The new features in these three areas are among the most anticipated ones in SQL Server 2019. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> SQL Server 2019's big data clusters are built on a number of different technologies, including SQL Server on Linux in Docker containers, Apache Spark, Hadoop and <a href="https://www.techtarget.com/searchitoperations/definition/Google-Kubernetes">Kubernetes</a>. Big data clusters on SQL Server 2019 enable users to deploy scalable clusters of SQL Server containers on Kubernetes that can read, write and process big data using Transact-SQL. A high-level overview of SQL Server 2019's big data clusters is shown in Figure 2. There, you see a big data cluster that consists of SQL Server and Spark Linux containers using Kubernetes for container management and orchestration. Queries are parallelized across multiple Docker containers running on a scalable group of nodes that comprise the compute tier. You can run advanced analytics and machine learning using Spark. Big data cluster activities are managed by a SQL Server master instance. <figure class="main-article-image half-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/editorial/121318_SQL_big-data-clusters_Fig1.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/editorial/121318_SQL_big-data-clusters_Fig1_half_column_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/editorial/121318_SQL_big-data-clusters_Fig1_half_column_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/editorial/121318_SQL_big-data-clusters_Fig1.png 1280w" alt="SQL Server 2019's big data clusters at a glance" height="209" width="279"> <figcaption> Figure 2. A high-level overview of SQL Server 2019's big data clusters. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> <h3>Database engine enhancements</h3> While the new big data clusters provide SQL Server 2019 with improved support for big data analytics, there's no doubt that the relational database engine is still the core of SQL Server 2019. Not surprisingly, the new release brings with it a number of new capabilities. SQL Server 2019 has a database <a href="https://docs.microsoft.com/en-us/sql/relational-databases/databases/view-or-change-the-compatibility-level-of-a-database?view=sql-server-2017" target="_blank" rel="noopener">compatibility level</a> of 150; compatibility level hints now dictate a query's exact compatibility level using the OPTION (USE) hint. In fact, lightweight query profiling infrastructure is enabled by default. New static data masking can sanitize sensitive data in copies of SQL Server databases. SQL Server 2019 also supports UTF-8 character encoding. SQL Server 2019's new batch mode over row store capability enables queries to process data in batches instead of one row at a time; earlier releases limited batch mode to columnstore queries. Resumable online index creation enables an index creation operation to be paused and then later resumed from where it left off instead of having to restart. <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> Microsoft is preparing to release SQL Server 2019, and it has added a surprising number of new capabilities in a short time. </figure> </div> </blockquote> In earlier versions of SQL Server, creating a clustered columnstore index was an offline operation that required all changes to the underlying tables stop until the CCI was created; SQL Server 2019 enables users to create CCIs online. Support for compression estimates of columnstore indexes can be obtained using: sp_estimate_data_compression_savings A new APPROX_COUNT_DISTINCT function uses statistics instead of reading all of the data to get a count of distinct values in a column; the outcome is estimated to be within 2% of the actual value. SQL Server 2019 also supports SQL Server Machine Learning Services failover clusters. Always Encrypted with secure enclaves enables SQL Server 2019 to perform in-place encryption and decryption actions without shifting the data out of the server. This enables data that is encrypted using Always Encrypted to be used in range scans, LIKE queries and aggregate functions. An enclave appears as a black box to the process that is calling it. Even though the data in the enclave is decrypted, its values cannot be seen by the calling process. <h3>Always On Availability Groups with Kubernetes</h3> SQL Server 2019 also includes several significant improvements to Always On Availability Groups (AGs). The biggest change is the ability to configure Always On AGs using Kubernetes as an orchestration layer in place of Windows failover clustering. Kubernetes can be used to create an AG group consisting of SQL Server 2019 on Docker containers that works like a shared disk Windows failover cluster instance. Technically, this not a new capability, but SQL Server 2019 enables enhanced instance health check monitoring using the same operator pattern as AG health checks. An overview of SQL Server AGs on Kubernetes is shown in Figure 3. <figure class="main-article-image half-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/editorial/121318_SQL_AlwaysOn-availability-groups_Fig2.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/editorial/121318_SQL_AlwaysOn-availability-groups_Fig2_half_column_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/editorial/121318_SQL_AlwaysOn-availability-groups_Fig2_half_column_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/editorial/121318_SQL_AlwaysOn-availability-groups_Fig2.png 1280w" alt="An overview of a SQL Server Always On Availability Group on Kubernetes" height="162" width="280"> <figcaption> Figure 3. A SQL Server Always On Availability Group on Kubernetes. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> Kubernetes clusters consist of a master node that is used for management and worker nodes that run pods. Each pod can have one or more related containers. A SQL Server AG can span multiple nodes and can provide automated failover for both node and pod failure. Always On AGs now support up to five synchronous replicas, an increase from the previous limit of three in SQL Server 2017. Synchronous replicas support automatic failover within the AG. In an AG with five synchronous replicas, there is one primary replica and four synchronous secondary replicas. Additionally, a new secondary-to-primary replica connection redirection capability enables client connections to be directed to the primary replica, regardless of the target server specified in the connection string without a listener. <h3>SQL Server on Linux and Docker container enhancements</h3> Since its initial introduction with SQL Server 2017, Microsoft has continued to bring SQL Server on Linux into closer parity with SQL Server on Windows. Some of the newest enhancements with the SQL Server 2019 release include support for transactional replication and distributed transactions. SQL Server 2019 on Linux instances can participate in transactional, merge and snapshot replication topologies as a publisher, distributor or subscriber. Support for Microsoft Distributed Transaction Coordinator (MSDTC) enables SQL Server on Linux instances to conduct distributed transactions. To enable this capability, Microsoft created a Linux version of MSDTC that runs within the SQL Server process. SQL Server 2019 provides improved integration between SQL Server on Linux and Active Directory (<a href="https://www.techtarget.com/searchwindowsserver/definition/Active-Directory">AD</a>). SQL Server on Linux instances can use AD to authenticate users, as well as for replication, distributed queries and AGs. Additionally, the Linux version of SQL Server 2019 includes OpenLDAP support for third-party AD providers. SQL Server 2019 on Linux also provides new support for in-database machine learning. Container support for SQL Server 2019 also has a couple of notable improvements. One is a new Red Hat Enterprise Linux (<a href="https://www.techtarget.com/searchdatacenter/definition/Red-Hat-Enterprise-Linux-RHEL">RHEL</a>)-certified Docker container image: docker pull mcr.microsoft.com/mssql/rhel/server:2019-CTP2.1 Microsoft has also created a new <a href="https://azure.microsoft.com/en-us/services/container-registry/" target="_blank" rel="noopener">master container registry</a>. This is intended to be the official container registry for the distribution of Microsoft product containers. Microsoft Container Registry does not have its own catalog and is meant to support existing catalogs, like Docker Hub, Red Hat Container Catalog and Azure Marketplace. To download images from the new location, use a command like the following Docker pull command: docker pull mcr.microsoft.com/mssql/server <h3>SQL Server 2019 management tools</h3> Like the previous versions of SQL Server, <a href="https://searchsqlserver.techtarget.com/definition/Microsoft-SQL-Server-Management-Studio-SSMS">SQL Server Management Studio</a>, SQL Server Data Tools and <a href="https://searchsqlserver.techtarget.com/definition/SQL-Operations-Studio">Azure Data Studio</a> are the primary management and development tools for SQL Server 2019. All of these tools are free downloads that are available separately from the main SQL Server 2019 release. They are developed and made available on a separate and more frequent release cycle than SQL Server. The new Azure Data Studio, which was known as SQL Operations Studio before Microsoft made it generally available in September 2018, is shown in Figure 4. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/editorial/121318_SQL_Azure-Studio_Fig3.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/editorial/121318_SQL_Azure-Studio_Fig3_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/editorial/121318_SQL_Azure-Studio_Fig3_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/editorial/121318_SQL_Azure-Studio_Fig3.png 1280w" alt="The new Azure Data Studio" height="320" width="560"> <figcaption> Figure 4. Azure Data Studio. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> The SQL Server 2019 release includes new big data integration features, a collection of database engine enhancements and improved Linux and container support. https://cdn.ttgtmedia.com/visuals/searchBusinessAnalytics/data_analytics/businessanalytics_article_013.jpg https://www.techtarget.com/searchdatamanagement/tip/SQL-Server-2019-improves-Linux-container-support Mon, 17 Dec 2018 15:53:00 GMT SQL Server 2019 improves Linux, container support The PowerShell Gallery is an online repository that features PowerShell module downloads and other code, but you'll need to be aware of some potential hang-ups associated with its use. The gallery and the PowerShellGet module are a subset of the <a href="https://www.techtarget.com/searchwindowsserver/Comprehensive-PowerShell-guide-for-new-and-seasoned-admins">PowerShell package management</a> system. You can view your currently installed repositories using the Get-PSRepository cmdlet. A PowerShell installation will only show the gallery. You work with the gallery with the <a href="https://github.com/powershell/powershellget" target="_blank" rel="noopener">PowerShellGet</a> module. You access software packages via the <a href="https://github.com/OneGet/oneget" target="_blank" rel="noopener">PackageManagement</a> module -- previously OneGet. Both modules are open source. New versions of <a href="https://www.techtarget.com/searchenterprisedesktop/tip/How-to-create-Windows-10-toast-notifications-with-PowerShell">PowerShell module downloads</a> are frequently published to the gallery, and modules need to be kept up to date for full functionality. For instance, without the latest version of the PowerShellGet module, you cannot publish modules to the gallery. Because the PowerShell Gallery is a public code repository, you should always test any code before using it. <div class="extra-info"> <div class="extra-info-inner"> <h4>My favorite modules</h4> There are more than 3,300 PowerShell module downloads available in the PowerShell Gallery; here are some worth a closer look. <ul class="default-list"> <li>ISESteroids. This module provides a set of extensions for the <a href="https://www.techtarget.com/searchwindowsserver/feature/Handy-PowerShell-ISE-add-ons-for-server-administrators">PowerShell Integrated Scripting Environment</a>. You'll need to buy a license from the author if you want to use it beyond the 10-day trial period. I've found it invaluable for catching coding issues, analyzing scripts and formatting code.</li> <li>Pester. This module runs a testing framework for PowerShell code and infrastructure configurations. Pester is also helpful for troubleshooting. Pester is installed with Windows PowerShell v5.1 -- it's not in <a href="https://www.techtarget.com/searchwindowsserver/tip/PowerShell-Core-61-offers-many-small-improvements">PowerShell Core from v6.1</a> onwards -- but newer versions are available from the PowerShell Gallery.</li> <li>PlatyPS. This module generates a PowerShell help file in XAML from a markdown file.</li> <li>PoshRSJob. This module uses PowerShell runspaces to provide parallel execution. A good replacement for workflows.</li> <li>Dbatools. This module provides a huge number of commands to automate SQL Server database development and administration.</li> </ul> </div> </div> The time you try to use the PowerShellGet cmdlets on a machine, you'll get a notice to install the NuGet provider: NuGet provider is required to continue PowerShellGet requires NuGet provider version '2.8.5.201' or newer to interact with NuGet-based repositories. The NuGet provider must be available in 'C:\Program Files\PackageManagement\ProviderAssemblies' or 'C:\Users\richard\AppData\Local\PackageManagement\ProviderAssemblies'. You can also install the NuGet provider by running 'Install-PackageProvider -Name NuGet -MinimumVersion 2.8.5.201 -Force'. Do you want PowerShellGet to install and import the NuGet provider now? [Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"): Press Y for the install to proceed. The <a href="https://www.techtarget.com/searchwindowsserver/tip/Azure-PowerShell-cmdlets-monitor-manage-VMs">PowerShellGet module</a> is one of a set of providers. Others include the following: <ul class="default-list"> <li>NuGet: a .NET package manager</li> <li>DockerProvider: offers access to container images</li> <li>Chocolatey: provides access to the Chocolatey software package management system</li> </ul> You can find the current list of possible providers using the following cmdlet: Find-PackageProvider <div class="imagecaption alignRight"> <iframe src="https://www.youtube.com/embed/kwsAcSP4P6Q" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen width="320" height="180" frameborder="0"></iframe> Import a module from the PowerShell Gallery into Azure Automation </div> A provider will access one or more sources, such as the PowerShell Gallery. You can view the available packages on NuGet using this command: Find-Package -Provider NuGet -Source https://www.nuget.org/api/v2 The focus of this article is fetching PowerShell code from the PowerShell Gallery using the PowerShellGet module. PowerShellGet comes with PowerShell, but it might require an update. The following table shows the PowerShellGet versions installed with the latest three versions of PowerShell. <table cellspacing="3" cellpadding="3" border="1"> <tbody> <tr> <td width="143">   </td> <td width="118"> Windows PowerShell 5.1.17134.228 </td> <td width="110"> PowerShell 6.1.0 </td> <td width="124"> PowerShell 6.2.0-preview.1 </td> <td width="105"> PowerShell Gallery </td> </tr> <tr> <td width="143"> PowerShellGet </td> <td width="118"> 1.6.0 </td> <td width="110"> 1.6.7 </td> <td width="124"> 2.0.1 </td> <td width="105"> 2.0.1 </td> </tr> <tr> <td width="143"> PackageManagement </td> <td width="118"> 1.1.7.0 </td> <td width="110"> 1.1.7.2 </td> <td width="124"> 1.2.2 </td> <td width="105"> 1.2.2 </td> </tr> </tbody> </table> <h3>Content offered by the PowerShell Gallery</h3> The gallery offers three types of code: <ul class="default-list"> <li>Modules, accessed with the Find-Module cmdlet.</li> <li><a href="https://www.techtarget.com/searchwindowsserver/definition/Microsoft-Windows-PowerShell-DSC-Desired-State-Configuration">Desired State Configuration (DSC)</a> resources, found with the Find-DscResource cmdlet, which shows individual resources, but you have to download the whole module that contains the resource.</li> <li>scripts, accessed with the Find-Script cmdlet.</li> </ul> The format of the command used to find the code types is Find-X, where X is the module, <a href="https://www.techtarget.com/searchwindowsserver/tip/Tap-PowerShell-DSC-resources-to-automate-web-server-construction">DSC resource</a> or script. Use the Find-Command cmdlet to search modules in the PowerShell Gallery for specific commands. The Find-RoleCapability <a href="https://www.techtarget.com/whatis/definition/cmdlet">cmdlet</a> searches for Just Enough Administration role capabilities. None were available at the time this article was published. <h3>Discover and install a PowerShell module</h3> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> Most users download the latest version of the module, but it's possible to supply a minimum, maximum or required version to download a specific version. </figure> </div> </blockquote> There is a lot of code available in the PowerShell Gallery. At last count, there were more than 3,000 modules to download. To fetch a module, use the PowerShell Community Extensions module. I know the name of the module -- PSCX -- but you can search the gallery on partial names: Find-Module -Name *community* Or the description: Find-Module | where Description -like "*PowerShell Community Extensions*" By default, a module search only shows the latest version. Find-Module -Name pscx This command shows version 3.3.2 as the current version. To see all available versions, use: Find-Module -Name pscx -AllVersions To show the beta versions, use this command: Find-Module -Name pscx -AllVersions -AllowPrerelease <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_Module-beta-versions_Fig1.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_Module-beta-versions_Fig1_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_Module-beta-versions_Fig1_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_Module-beta-versions_Fig1.png 1280w" alt="Module beta versions" height="252" width="559"> <figcaption> The -AllowPrerelease parameter shows any beta versions of the module in the PowerShell Gallery. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> Most users download the latest version of the module, but it's possible to supply a minimum, maximum or required version to download a specific version. To install a module, use this command: Install-Module -Name pscx The PowerShell Gallery is a public repository and, as such, is untrusted. Before the install occurs, PowerShell issues a warning and asks if you want to continue. You can avoid these messages and set the PowerShell Gallery to be a trusted repository, but I don't recommend it. The progress bar will indicate what is happening during the download and install process. Also, PSCX contains some commands that have the same name as commands in PowerShell, so you'll get an error. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_PowerShell-warning_Fig2.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_PowerShell-warning_Fig2_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_PowerShell-warning_Fig2_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_PowerShell-warning_Fig2.png 1280w" alt="PowerShell warning" height="220" width="559"> <figcaption> PowerShell will issue a warning if you attempt to install a module from the PowerShell Gallery because it is a public repository. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> To import the commands in the module, use the -AllowClobber parameter: Install-Module -Name pscx -AllowClobber You won't get any feedback to indicate if the installation was completed. Run the following command to show if the module was installed in the C:\Program Files\WindowsPowerShell\Modules folder: Get-Module -ListAvailable pscx On machines that have Windows PowerShell v5.1 and <a href="https://www.techtarget.com/searchwindowsserver/tip/Mind-the-feature-gaps-in-the-PowerShell-open-source-project">PowerShell Core v6.x installed</a>, you'll need to install the module twice -- once for each version -- because PowerShell Core stores downloaded modules in the C:\Program Files\PowerShell\Modules folder. Don't be tempted to add the Windows PowerShell v5.1 folder to the PowerShell v6 module path or vice versa. The PowerShell Gallery installation process works in a similar fashion to PowerShell Core. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_PowerShell-Core-module-installation_Fig3.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_PowerShell-Core-module-installation_Fig3_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_PowerShell-Core-module-installation_Fig3_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_PowerShell-Core-module-installation_Fig3.png 1280w" alt="PowerShell Core module installation" height="321" width="559"> <figcaption> On machines with both Windows PowerShell and PowerShell Core installed, the module needs to be installed twice. In this screenshot, the module installation is halted until the user grants access. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> <h3>Set the scope of the module installation</h3> You can control where the module installs with the -Scope parameter on Install-Module. You have two choices: <ul class="default-list"> <li>The AllUsers option installs the module in the C:\Program Files\WindowsPowerShell\Modules. folder -- Windows PowerShell -- or the C:\Program Files\PowerShell\Modules -- PowerShell Core -- folder.</li> <li>The CurrentUser option installs the module in the $home\Documents\WindowsPowerShell\Module folder -- Windows PowerShell -- or the $home\Documents\PowerShell\Modules folder -- PowerShell Core.</li> </ul> Without a defined scope, the defaults are: <ul class="default-list"> <li>Elevated session: scope defaults to AllUsers.</li> <li>Non-elevated session using PowerShellGet 2.0 and later: scope defaults to CurrentUser.</li> <li>Non-elevated session using PowerShellGet 1.6.7 and earlier: scope is undefined and installation fails.</li> </ul> My preference is to install with AllUsers because it results in a single copy of the module on the system, which makes it easier to keep track of what is installed using the Get-InstalledModule cmdlet. <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_Installed-modules_Fig4.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_Installed-modules_Fig4_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_Installed-modules_Fig4_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/editorial/121218_SWS_Installed-modules_Fig4.png 1280w" alt="Installed modules" height="213" width="559"> <figcaption> The Get-InstalledModule cmdlet shows the modules installed on the system and their respective version numbers. </figcaption> <div class="main-article-image-enlarge"> </div> </figure> <h3>Distributing modules to multiple machines</h3> In a situation that requires the installation of a module from the PowerShell Gallery on many machines, the best option is to create a local gallery, download the module and republish it to the local gallery, from which you can then install the module. This is also a good technique to distribute <a href="https://www.techtarget.com/searchitoperations/tutorial/How-to-create-PowerShell-modules-with-a-Plaster-template">your own modules</a> throughout your organization. Removing a module is a simple matter of uninstalling it with the following command: Uninstall-Module pscx -Force As a new version of a module gets released to the PowerShell Gallery, you'll want to upgrade your installed version. For instance, I have version 0.7.6 of the PlatyPS module on my machine, but checking for an update shows version 0.12 is available in the gallery. To update this installed module, I used the Update-Module cmdlet: Update-Module -Name platyPS If you try to update a module that was installed as part of the PowerShell installation, the following error occurs: Update-Module -Name PowerShellGet Update-Module : Module 'PowerShellGet' was not installed by using Install-Module, so it cannot be updated. At line:1 char:1 + Update-Module -Name PowerShellGet + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo          : InvalidOperation: (PowerShellGet:String) [Write-Error], WriteErrorException + FullyQualifiedErrorId : ModuleNotInstalledUsingInstallModuleCmdlet,Update-Module The way around this is to force the installation of the module: Install-Module -Name PowerShellGet -AllowClobber -Force The newly installed version of the module will then be used by default. You can then use Update-Module to perform future updates when new versions arrive in the PowerShell Gallery. Learn how to get the most out of the PowerShell Gallery with some distinctions for administrators who use Windows PowerShell or PowerShell Core. https://cdn.ttgtmedia.com/visuals/searchSoftwareQuality/models_methodologies/softwarequality_article_004.jpg https://www.techtarget.com/searchwindowsserver/tutorial/Working-with-PowerShell-module-downloads-from-the-gallery Thu, 13 Dec 2018 13:42:00 GMT Working with PowerShell module downloads from the gallery Vendors almost puff up as they make assertions about their wares, but the proof usually plays out a little further in the future. Claims from database giants<a href="https://www.techtarget.com/searchcio/news/252464886/In-depth-look-at-what-the-Oracle-Microsoft-agreement-means-to-CIOs"> Microsoft and Oracle</a> for their respective data platforms were tested at recent conferences focused partly on their database technologies, according to TechTarget reporters back from the Oracle OpenWorld 2018 and PASS Summit 2018 events. The PASS conference was notable for word on Microsoft Azure SQL Database developments. It is due to be updated on Dec. 1 in the form of a high-end release of the Microsoft Azure SQL Database Managed Instance cloud service, which debuted in a general-purpose version last month. The Business Critical release, the company now admits, addresses restrictions that have put Microsoft's cloud database at a disadvantage versus on-premises SQL Server. This time, it is more nearly equivalent on functionality in comparison to the venerable on-premises database technology. In this podcast, Craig Stedman, senior executive editor for SearchDataManagement and TechTarget's other data and analytics websites, outlines the progress claimed for Microsoft Azure SQL Database, which includes support for increased database sizes that will put it more on par with the large data stores that are usually found today. Reporting on the Oracle end of the spectrum, Jack Vaughan, senior news writer for SearchDataManagement, said in the podcast that Oracle is placing significant attention on Oracle Cloud at Customer as a way to prepare on-premises Oracle databases for the <a href="https://www.techtarget.com/searchaws/news/252472355/Amazon-wraps-Oracle-to-AWS-migration-looks-to-poach-workloads">eventual move to cloud</a>. Meanwhile, data managers and their corporate counterparts at OpenWorld were scrutinizing the vendor's <a href="https://www.techtarget.com/searchoracle/news/252451377/Users-look-to-Oracle-Autonomous-Database-to-lighten-IT-load">autonomous cloud services</a> based on Oracle Database 18c -- with <a href="https://www.techtarget.com/searchoracle/feature/Key-new-features-coming-in-Oracle-Database-19c">the upcoming Oracle 19c</a> to follow -- as they together plotted their courses to the cloud. Automated indexing is next on the docket for the Oracle Autonomous Database cloud technology -- although, as was evident at the conference, its advance will be somewhat incremental. The podcasters agreed it will take some time for database administrators (DBAs) to formally sign off on the cloud version of the Oracle database. Many DBAs harbor some worries about Autonomous Database's <a href="ttps://searchoracle.techtarget.com/feature/What-Oracle-performance-tuning-tools-can-and-cant-do-for-DBAs">auto-tuning and performance monitoring</a> capabilities. In the end, the podcast discussion naturally turned to Amazon and its AWS cloud platform. Amazon's success in the cloud has, in no small part, spurred Oracle and Microsoft cloud efforts, both database-oriented and other. A new Microsoft Azure SQL Database Managed Instance service seeks to span the gap between cloud and on-premises systems. Meanwhile, Oracle also has cloud plans of the data kind. https://www.techtarget.com/searchdatamanagement/podcast/Microsoft-Azure-SQL-Database-and-Oracle-19c-start-to-rev Tue, 27 Nov 2018 12:30:00 GMT Microsoft Azure SQL Database and Oracle 19c start to rev Docker containers are great for <a href="https://www.techtarget.com/searchdatamanagement/definition/SQL-Server">SQL Server</a> development and test scenarios, as they allow database administrators and developers to deploy SQL Server instances within seconds. Users are able to spin up a container complete with SQL Server and attached databases as quickly as they can type in a docker run command. Containers can also be used in <a href="https://searchsqlserver.techtarget.com/feature/How-SQL-Server-containers-can-help-ease-database-deployment">production SQL Server applications</a>. However, there are some things you should know about deploying SQL Server containers before you start to use them. Here are answers to five frequently asked questions on running SQL Server in <a href="https://www.techtarget.com/searchitoperations/answer/Why-is-Dockers-container-approach-so-important">Docker containers</a>. Q: Can I use SQL Server Management Studio (<a href="https://searchsqlserver.techtarget.com/definition/Microsoft-SQL-Server-Management-Studio-SSMS">SSMS</a>) or Azure Data Studio to connect to SQL Server in a container and run queries? Yes. You can connect to a SQL Server container to query and manage it in three ways; two of them can be done with SSMS or <a href="https://searchsqlserver.techtarget.com/definition/SQL-Operations-Studio">Azure Data Studio</a>, which was called SQL Operations Studio before Microsoft made it generally available in September 2018. The three options are: <ul class="default-list"> <li>Connecting from within the container itself. You need to use the sqlcmd command-line tool to do this because Docker containers don't support a GUI. The easiest way is to run a docker exec command with the notation for the server name. For example: docker exec -it dab sqlcmd -S. -Usa.</li> <li>Connecting from the <a href="https://www.techtarget.com/searchitoperations/tip/Build-a-secure-Docker-host-environment-on-Linux-systems">Docker host</a>. To do this, you can use sqlcmd or a management tool like SSMS or Azure Data Studio. You simply enter the IP address of the container, as here: Server = 172.17.0.2.</li> <li>Connecting from an external system. You can also use sqlcmd, SSMS or Azure Data Studio to connect to a SQL Server container from a host system that's separate from the Docker host by entering the Docker host's IP address followed by a comma and the container's TCP port mapping. For example: Server = 192.168.100.123,1403.</li> </ul> Note that when you start a container, you have the option of mapping it to SQL Server's default port 1433 or another port that's exposed on the host. If you want to run multiple SQL Server containers on the same host, you need to map them to different ports to avoid conflicts. Q: How can I attach external databases to SQL Server containers? Docker containers are stateless. While that can work for development and test uses with a system like SQL Server, in production applications, you typically need to persist databases and the changes made to the data in them. Docker lets users do so by attaching database files stored externally to a container that's running SQL Server. The external repositories are maintained separately from the container so the databases remain in place if it's stopped and deleted. There are two ways to <a href="https://searchsqlserver.techtarget.com/tip/2-ways-to-attach-SQL-Server-database-files-to-Linux-containers">attach external databases to containers</a>. One is to use Docker data volumes, which reside in the Docker-managed portion of the host machine's file system. Volumes are the preferred method of enabling data persistence in containers, according to the <a href="https://docs.docker.com/storage/volumes/" target="_blank" rel="noopener">online documentation</a> for the Docker technology. First, use the docker volume create command to set up a volume. You can then mount the volume to a container using the following command: docker run -d -p 1433:1433 -e ACCEPT_EULA=Y -e MSSQL_SA_PASSWORD=SQLpwd20017 -v MSSQLDBvol:/var/opt/mssql microsoft/mssql-server-linux <figure class="main-article-image half-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/editorial/081618_SQL_Operations_Figure3.png"> <img data-src="https://searchsqlserver.techtarget.com/rms/editorial/081618_SQL_Operations_Figure3_half_column_mobile.png" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/editorial/081618_SQL_Operations_Figure3_half_column_mobile.png 960w,https://searchsqlserver.techtarget.com/rms/editorial/081618_SQL_Operations_Figure3.png 1280w" alt="Screenshot of interface to restore a SQL Server backup file" data-credit="TechTarget" height="193" width="280"> <figcaption> Interface to attach a SQL Server database to a container by restoring a backup file </figcaption> <div class="main-article-image-enlarge"> </div> </figure> Volumes can be used with SQL Server in both Windows and Linux systems, but they work differently in the two OSes. In Linux, you map SQL Server's default data directory to one that Docker creates in a volume and Docker copies the system's database files into the volume. To add more databases, copy their data and log files to the volume and attach them to the container. Otherwise, you can copy a database backup file to the volume and restore it, which makes the database available to the container. That isn't the case with Windows-based volumes. With Docker on Windows, you can only map to empty directories, and only user database files can be present in a volume. Also, databases need to be attached to containers via a docker run command. The other way to attach external databases is to mount a file or directory that's on the Docker host in the container. Called a bind mount, this process is also somewhat different for SQL Server containers on Windows and Linux. You can directly attach a folder that contains a database's data files and transaction log files to a SQL Server container running on Windows, as you can see in the following example: docker run -d -p 1433:1433 -e sa_password=SQLpwd2017 -e ACCEPT_EULA=Y -v C:/temp/:C:/temp/ -e attach_dbs="[{'dbName':'AdventureWorks2014','dbFiles':['C:\\temp\\AdventureWorks2014_Data.mdf', 'C:\\temp\\AdventureWorks2014_Log.ldf']}]" microsoft/mssql-server-windows-express SQL Server on Linux supports mounting a folder on the Docker host to the default SQL Server data directory in a container. Here's an example of how to do that, after which you can attach the database files to the container by following the steps listed above on how to attach files in a volume to a Linux-based container: docker run -d -p 1433:1433 -e ACCEPT_EULA=Y -e SA_PASSWORD=SQLpwd2017 -v /home/administrator/MSSQLDBs:/var/opt/mssql microsoft/mssql-server-linux Q: How are SQL Server containers licensed? The licensing for SQL Server in containers is similar to running it in a virtual machine (<a href="https://searchservervirtualization.techtarget.com/definition/virtual-machine">VM</a>). Containers differ from VMs in that all of the containers running on a physical host share the operating system and other resources. This can have financial benefits, as you require fewer OS licenses, but each instance of SQL Server must still be licensed. Under Microsoft's Server + CAL licensing model, which can be used with the Standard edition of SQL Server, you need to purchase one SQL Server license for every container running a SQL Server instance. Also, each user or device that accesses an instance will require a SQL Server 2017 client access license (CAL). <figure class="main-article-image full-col" data-img-fullsize="https://searchsqlserver.techtarget.com/rms/onlineImages/microservices-containers_vs_vms.jpg"> <img data-src="https://searchsqlserver.techtarget.com/rms/onlineImages/microservices-containers_vs_vms_mobile.jpg" class="lazy" data-srcset="https://searchsqlserver.techtarget.com/rms/onlineImages/microservices-containers_vs_vms_mobile.jpg 960w,https://searchsqlserver.techtarget.com/rms/onlineImages/microservices-containers_vs_vms.jpg 1280w" alt="Key differences between virtual machines and Docker containers" data-credit="TechTarget" height="197" width="520"> <figcaption> How virtual machines and containers differ from one another </figcaption> <div class="main-article-image-enlarge"> </div> </figure> Under the per-core licensing model available for both the Standard and Enterprise editions, all the virtual cores in the container running an instance of SQL Server must be licensed, and there's a four-core license minimum per container. Additional licenses can be purchased in packs of two. With SQL Server 2017 Enterprise Edition, customers who have licensed the physical cores on a server instead of using a virtual OS environment can run a number of containers equal to the number of core licenses assigned to the server. Q: How can I control CPU assignments for SQL Server containers? Unlike VMs, where you explicitly configure the number of virtual CPUs in them, containers, by default, share all of the host resources -- meaning that if the host has 16 CPUs, each container will have 16 virtual CPUs. While that's good for performance, it can also have unintended licensing ramifications. However, you can limit the virtual CPUs in a SQL Server container by using the --cpus=<value> parameter as part of the docker run command to specify how many of the available CPUs the container can utilize. For example: Docker run -d -p 1433:1433 --cpus=1 -e sa_password=PwdSQL2017 -e ACCEPT_EULA=Y microsoft/mssql-server-windows-express In Windows, you can also use the -cpu-count parameter to do the same thing. In addition, you can use --cpuset-cpus in both Linux and Windows to specify the particular CPUs that will be used to execute processing jobs. You can likewise limit the maximum amount of memory a container can use via the -m or --memory=<value> parameter; the minimum allowed value is 4 MB. Q: How do I troubleshoot SQL Server running in a Docker container? Docker containers typically run in detached mode, which makes them a background task -- meaning that you don't have any visual feedback when something goes wrong. Here are the three biggest issues you may need to troubleshoot when you start running SQL Server containers, and how to do so: <ul class="default-list"> <li>A container won't run or only runs for a couple of seconds. If this happens, use the docker logs command to examine the container logs. If that doesn't help, try the docker run command with the -it parameter in place of the -d parameter. Doing so will show the container activity, hopefully allowing you to see what's wrong.</li> <li>You can't connect to SQL Server in the container. If that's the case, use the docker inspect command to review the container's IP address and port. If you're trying to connect from an external host, be sure that you're using a comma between the Docker host's IP address and the container's mapped SQL Server port. For example: 192.168.100.123,1403.</li> <li>You can't attach an external database to the container. First, use docker logs to check for any startup errors. Next, verify the host path and the names of the database files in the external directory. If you're using Windows-based containers, make sure the JSON string used in the attach_dbs parameter is correctly formatted.</li> </ul> Running SQL Server in containers creates new challenges for database administrators. The answers to these questions can guide you through some of them. https://cdn.ttgtmedia.com/visuals/searchSQLServer/microsoft_server_database_dev/sqlserver_article_006.jpg https://www.techtarget.com/searchdatamanagement/tip/5-FAQs-on-SQL-Server-containers-and-how-to-manage-them Thu, 15 Nov 2018 16:25:00 GMT 5 FAQs on SQL Server containers and how to manage them SearchSQLServer Resources and Information from TechTarget 60 webmaster@techtarget.com